[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:07 dist-test-slave-7s64 krb5kdc[396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:07 dist-test-slave-7s64 krb5kdc[396](info): set up 2 sockets
Jun 20 15:16:07 dist-test-slave-7s64 krb5kdc[396](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 15:16:08.832166 389 mini_kdc.cc:121] Time spent starting KDC: real 2.321s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 15:16:09.047000 389 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.214s user 0.001s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:09 dist-test-slave-7s64 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:16:09.181885 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34467
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:34467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:09.486910 409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:09.487164 409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:09.487323 409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:09.494486 409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:09.494575 409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:09.494669 409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:09.494769 409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:09.494836 409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:09.503926 409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:34467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:09.505075 409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:09.506347 409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:09.518095 417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:09.518169 415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:09.518201 414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:09.518731 409 server_base.cc:1034] running on GCE node
I20260620 15:16:09.519677 409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:09.520956 409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:09.522140 409 hybrid_clock.cc:648] HybridClock initialized: now 1781968569522066 us; error 62 us; skew 500 ppm
I20260620 15:16:09.522456 409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:09 dist-test-slave-7s64 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:09.525796 409 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:16:09.527084 409 webserver.cc:460] Webserver started at http://127.0.97.126:43345/ using document root <none> and password file <none>
I20260620 15:16:09.527704 409 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:09.527835 409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:09.528111 409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:09.530866 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "c83c99ac214a4e908ef311878e2e3da6"
format_stamp: "Formatted at 2026-06-20 15:16:09 on dist-test-slave-7s64"
server_key: "07fe7d8ee8e3c8dda116a31ee1cff322"
server_key_iv: "4571f2a95ba7a457fc84de4d9147dfec"
server_key_version: "encryptionkey@0"
I20260620 15:16:09.531575 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "c83c99ac214a4e908ef311878e2e3da6"
format_stamp: "Formatted at 2026-06-20 15:16:09 on dist-test-slave-7s64"
server_key: "07fe7d8ee8e3c8dda116a31ee1cff322"
server_key_iv: "4571f2a95ba7a457fc84de4d9147dfec"
server_key_version: "encryptionkey@0"
I20260620 15:16:09.536247 409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 15:16:09.539459 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:09.540692 409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 15:16:09.540859 409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "c83c99ac214a4e908ef311878e2e3da6"
format_stamp: "Formatted at 2026-06-20 15:16:09 on dist-test-slave-7s64"
server_key: "07fe7d8ee8e3c8dda116a31ee1cff322"
server_key_iv: "4571f2a95ba7a457fc84de4d9147dfec"
server_key_version: "encryptionkey@0"
I20260620 15:16:09.541029 409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:09.562768 409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:09.566946 409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:09.567235 409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:09.586169 409 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:34467
I20260620 15:16:09.586206 475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:34467 every 8 connection(s)
I20260620 15:16:09.587474 409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:09.591053 476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:09.595103 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 409
I20260620 15:16:09.595381 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:09.597724 389 external_mini_cluster.cc:1428] Setting key 2dd457a4c2c9e2f78b3c8934cbe5d908
I20260620 15:16:09.603106 476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6: Bootstrap starting.
I20260620 15:16:09.606104 476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:09.607004 476 log.cc:824] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:09.609499 476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6: No bootstrap required, opened a new log
I20260620 15:16:09.618786 476 raft_consensus.cc:348] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83c99ac214a4e908ef311878e2e3da6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34467 } }
I20260620 15:16:09.619164 476 raft_consensus.cc:374] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:09.619272 476 raft_consensus.cc:729] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83c99ac214a4e908ef311878e2e3da6, State: Initialized, Role: FOLLOWER
I20260620 15:16:09.619868 476 consensus_queue.cc:260] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83c99ac214a4e908ef311878e2e3da6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34467 } }
I20260620 15:16:09.620072 476 raft_consensus.cc:388] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:09.620239 476 raft_consensus.cc:482] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:09.620393 476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:09.622951 476 raft_consensus.cc:504] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83c99ac214a4e908ef311878e2e3da6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34467 } }
I20260620 15:16:09.623462 476 leader_election.cc:302] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c83c99ac214a4e908ef311878e2e3da6; no voters:
I20260620 15:16:09.624403 476 leader_election.cc:288] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:09.624557 481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:09.626225 476 sys_catalog.cc:564] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:09.628471 481 raft_consensus.cc:686] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [term 1 LEADER]: Becoming Leader. State: Replica: c83c99ac214a4e908ef311878e2e3da6, State: Running, Role: LEADER
Jun 20 15:16:09 dist-test-slave-7s64 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:09.629063 481 consensus_queue.cc:237] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83c99ac214a4e908ef311878e2e3da6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34467 } }
I20260620 15:16:09.632421 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c83c99ac214a4e908ef311878e2e3da6. Latest consensus state: current_term: 1 leader_uuid: "c83c99ac214a4e908ef311878e2e3da6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83c99ac214a4e908ef311878e2e3da6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34467 } } }
I20260620 15:16:09.632779 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:09.632743 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c83c99ac214a4e908ef311878e2e3da6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83c99ac214a4e908ef311878e2e3da6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34467 } } }
I20260620 15:16:09.633093 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:09.640799 493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:09.643567 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:09.602199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44166 (local address 127.0.97.126:34467)
0620 15:16:09.603024 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:09.603038 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:09.604708 (+ 1670us) server_negotiation.cc:407] Connection header received
0620 15:16:09.614389 (+ 9681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:09.614414 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:09.614873 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:09.615309 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:09.617791 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:09.619706 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:09.621514 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:09.622515 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:09.622696 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:09.630776 (+ 8080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:09.630790 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:09.630818 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:09.630880 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:09.637501 (+ 6621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:09.638777 (+ 1276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:09.638788 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:09.638799 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:09.639031 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:09.639714 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:09.639722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:09.639728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:09.640467 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:09.641457 (+ 990us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:09.641760 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 15:16:09.642141 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":358,"threads_started":1}
I20260620 15:16:09.646099 493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:09.653532 493 catalog_manager.cc:1347] Generated new cluster ID: 38fa0ee5609b451aaffa8e4bd1773c8d
I20260620 15:16:09.653627 493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:09.664855 493 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:09.666070 493 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:09.674736 493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c83c99ac214a4e908ef311878e2e3da6: Generated new TSK 0
I20260620 15:16:09.675495 493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:16:09.715895 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34467
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:10.036242 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:10.036574 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:10.036746 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:10.043756 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:10.044267 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:10.044457 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:10.053723 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34467
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:10.054889 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:10.056102 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:10.068768 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:10.068838 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:10.068815 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:10.069592 502 server_base.cc:1034] running on GCE node
I20260620 15:16:10.070098 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:10.071015 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:10.072202 502 hybrid_clock.cc:648] HybridClock initialized: now 1781968570072170 us; error 26 us; skew 500 ppm
I20260620 15:16:10.072525 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:10 dist-test-slave-7s64 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:10.075335 502 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:16:10.076683 502 webserver.cc:460] Webserver started at http://127.0.97.65:42335/ using document root <none> and password file <none>
I20260620 15:16:10.077301 502 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:10.077430 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:10.077697 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:10.080291 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "21be61468c35412992f5e142fb2b8970"
format_stamp: "Formatted at 2026-06-20 15:16:10 on dist-test-slave-7s64"
server_key: "f146316d0791f4a7efad678ac41e299c"
server_key_iv: "47a6b5748775e56d7089dee4dd483b92"
server_key_version: "encryptionkey@0"
I20260620 15:16:10.081070 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "21be61468c35412992f5e142fb2b8970"
format_stamp: "Formatted at 2026-06-20 15:16:10 on dist-test-slave-7s64"
server_key: "f146316d0791f4a7efad678ac41e299c"
server_key_iv: "47a6b5748775e56d7089dee4dd483b92"
server_key_version: "encryptionkey@0"
I20260620 15:16:10.086024 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:16:10.089133 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:10.090370 502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:10.090523 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "21be61468c35412992f5e142fb2b8970"
format_stamp: "Formatted at 2026-06-20 15:16:10 on dist-test-slave-7s64"
server_key: "f146316d0791f4a7efad678ac41e299c"
server_key_iv: "47a6b5748775e56d7089dee4dd483b92"
server_key_version: "encryptionkey@0"
I20260620 15:16:10.090696 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:10.110644 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:10.114989 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:10.115245 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:10.116078 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:10.117278 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:10.117378 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:10.117506 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:10.117573 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:10.151882 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36961
I20260620 15:16:10.151957 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36961 every 8 connection(s)
I20260620 15:16:10.153093 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:10.160683 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 502
I20260620 15:16:10.160861 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:10.161563 389 external_mini_cluster.cc:1428] Setting key db6c1b472dbbde8dc5874da0ee3403b6
Jun 20 15:16:10 dist-test-slave-7s64 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:10.173259 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:10.155454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49905 (local address 127.0.97.126:34467)
0620 15:16:10.155835 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:10.155839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:10.156812 (+ 973us) server_negotiation.cc:407] Connection header received
0620 15:16:10.158488 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:10.158492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:10.158559 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:10.158655 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:10.159982 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:10.161140 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:10.162673 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:10.163134 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:10.163240 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:10.167121 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:10.167130 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:10.167137 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:10.167180 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:10.170043 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:10.170959 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:10.170963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:10.170965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:10.171078 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:10.171518 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:10.171521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:10.171523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:10.171797 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:10.171909 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:10.172819 (+ 910us) server_negotiation.cc:299] Negotiation successful
0620 15:16:10.172967 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":190,"threads_started":1}
I20260620 15:16:10.174619 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:10.155981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34467 (local address 127.0.97.65:49905)
0620 15:16:10.156670 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:10.156705 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:10.158241 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:10.158836 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:10.158843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:10.159325 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:10.159839 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:10.159858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:10.161282 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:10.161286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:10.162541 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:10.162554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:10.163344 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:10.163352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:10.163553 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:10.164634 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:10.164654 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:10.166892 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:10.170172 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:10.170183 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:10.170199 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:10.170826 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:10.171182 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:10.171189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:10.171195 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:10.171412 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:10.171925 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:10.171935 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:10.172381 (+ 446us) client_negotiation.cc:769] Sending connection context
0620 15:16:10.172696 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 15:16:10.173234 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":247,"threads_started":1}
I20260620 15:16:10.176100 630 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34467
I20260620 15:16:10.176383 630 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:10.177059 630 heartbeater.cc:511] Master 127.0.97.126:34467 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:16:10.178972 441 ts_manager.cc:194] Registered new tserver with Master: 21be61468c35412992f5e142fb2b8970 (127.0.97.65:36961)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260620 15:16:10.180626 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:49905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:16:10.198944 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34467
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:10.513520 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:10.513799 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:10.513965 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:10.520852 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:10.520983 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:10.521144 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:10.530747 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34467
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:10.531891 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:10.533105 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:10.545696 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:10.545746 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:10.546238 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:10.546411 636 server_base.cc:1034] running on GCE node
I20260620 15:16:10.546978 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:10.547727 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:10.548938 636 hybrid_clock.cc:648] HybridClock initialized: now 1781968570548899 us; error 49 us; skew 500 ppm
I20260620 15:16:10.549218 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:10 dist-test-slave-7s64 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:10.551987 636 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:16:10.553434 636 webserver.cc:460] Webserver started at http://127.0.97.66:34785/ using document root <none> and password file <none>
I20260620 15:16:10.554041 636 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:10.554157 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:10.554455 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:10.557248 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "372b661d328448f2852c618dda67d7e4"
format_stamp: "Formatted at 2026-06-20 15:16:10 on dist-test-slave-7s64"
server_key: "30eafaa062f5d327de04c24f0f232c90"
server_key_iv: "dea403cf22a83ebcb9f43e812ede94c6"
server_key_version: "encryptionkey@0"
I20260620 15:16:10.557960 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "372b661d328448f2852c618dda67d7e4"
format_stamp: "Formatted at 2026-06-20 15:16:10 on dist-test-slave-7s64"
server_key: "30eafaa062f5d327de04c24f0f232c90"
server_key_iv: "dea403cf22a83ebcb9f43e812ede94c6"
server_key_version: "encryptionkey@0"
I20260620 15:16:10.562736 636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:16:10.566323 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:10.567551 636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:16:10.567708 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "372b661d328448f2852c618dda67d7e4"
format_stamp: "Formatted at 2026-06-20 15:16:10 on dist-test-slave-7s64"
server_key: "30eafaa062f5d327de04c24f0f232c90"
server_key_iv: "dea403cf22a83ebcb9f43e812ede94c6"
server_key_version: "encryptionkey@0"
I20260620 15:16:10.567875 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:10.600216 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:10.605674 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:10.605967 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:10.607035 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:10.608529 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:10.608616 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:10.608721 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:10.608785 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:10.643859 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:34075
I20260620 15:16:10.644026 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:34075 every 8 connection(s)
I20260620 15:16:10.645170 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:10.653244 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 636
I20260620 15:16:10.653406 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:10.653870 389 external_mini_cluster.cc:1428] Setting key 1ac0d08a48dff90df42ee865250906ba
Jun 20 15:16:10 dist-test-slave-7s64 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:10.664461 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:10.647439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:60789 (local address 127.0.97.126:34467)
0620 15:16:10.647602 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:10.647607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:10.649165 (+ 1558us) server_negotiation.cc:407] Connection header received
0620 15:16:10.651181 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:10.651187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:10.651309 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:10.651433 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:10.652875 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:10.653693 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:10.655387 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:10.655771 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:10.655905 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:10.659091 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:10.659095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:10.659099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:10.659127 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:10.661646 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:10.662336 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:10.662340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:10.662342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:10.662458 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:10.662860 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:10.662865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:10.662868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:10.663119 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:10.663265 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:10.663931 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 15:16:10.664135 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:16:10.665412 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:10.647932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34467 (local address 127.0.97.66:60789)
0620 15:16:10.649018 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:10.649056 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:10.650851 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:10.651637 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:10.651645 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:10.652119 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:10.652729 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:10.652746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:10.653847 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:10.653855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:10.655246 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:10.655257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:10.655982 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:10.655990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:10.656213 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:10.656871 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:10.656892 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:10.658889 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:10.661796 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:10.661804 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:10.661818 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:10.662211 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:10.662562 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:10.662567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:10.662571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:10.662749 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:10.663245 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:10.663253 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:10.663559 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 15:16:10.663822 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 15:16:10.664300 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":257,"threads_started":1}
I20260620 15:16:10.667099 764 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34467
I20260620 15:16:10.667367 764 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:10.668000 764 heartbeater.cc:511] Master 127.0.97.126:34467 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260620 15:16:10.669356 441 ts_manager.cc:194] Registered new tserver with Master: 372b661d328448f2852c618dda67d7e4 (127.0.97.66:34075)
I20260620 15:16:10.670117 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:60789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:16:10.689941 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34467
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:11.003080 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:11.003329 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:11.003806 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:11.011111 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:11.011214 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:11.011325 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:11.020594 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34467
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:11.021705 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:11.022976 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:11.035934 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:11.036054 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:11.036023 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:11.036600 769 server_base.cc:1034] running on GCE node
I20260620 15:16:11.037107 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:11.037894 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:11.039072 769 hybrid_clock.cc:648] HybridClock initialized: now 1781968571039018 us; error 55 us; skew 500 ppm
I20260620 15:16:11.039320 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:11.042755 769 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:16:11.044025 769 webserver.cc:460] Webserver started at http://127.0.97.67:37793/ using document root <none> and password file <none>
I20260620 15:16:11.044744 769 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:11.044836 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:11.045064 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:11.047823 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "63c84f438b1a43c388ab01f1b13d3c69"
format_stamp: "Formatted at 2026-06-20 15:16:11 on dist-test-slave-7s64"
server_key: "b90898c2112b15bb11418d0b71ad59bc"
server_key_iv: "0f0263c08f8f204b280c2ee45460e034"
server_key_version: "encryptionkey@0"
I20260620 15:16:11.048620 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "63c84f438b1a43c388ab01f1b13d3c69"
format_stamp: "Formatted at 2026-06-20 15:16:11 on dist-test-slave-7s64"
server_key: "b90898c2112b15bb11418d0b71ad59bc"
server_key_iv: "0f0263c08f8f204b280c2ee45460e034"
server_key_version: "encryptionkey@0"
I20260620 15:16:11.053290 769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:16:11.056336 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:11.057590 769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 15:16:11.057762 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "63c84f438b1a43c388ab01f1b13d3c69"
format_stamp: "Formatted at 2026-06-20 15:16:11 on dist-test-slave-7s64"
server_key: "b90898c2112b15bb11418d0b71ad59bc"
server_key_iv: "0f0263c08f8f204b280c2ee45460e034"
server_key_version: "encryptionkey@0"
I20260620 15:16:11.057925 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:11.080879 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:11.084911 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:11.085175 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:11.086019 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:11.087217 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:11.087301 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:11.087462 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:11.087512 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:11.120152 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39645
I20260620 15:16:11.120191 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39645 every 8 connection(s)
I20260620 15:16:11.121753 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:11.124354 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 769
I20260620 15:16:11.124557 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:11.125154 389 external_mini_cluster.cc:1428] Setting key 9322b2e83b013f913b6ba7215b877396
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:11.143095 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:11.124057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54525 (local address 127.0.97.126:34467)
0620 15:16:11.124285 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:11.124293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:11.125543 (+ 1250us) server_negotiation.cc:407] Connection header received
0620 15:16:11.128090 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:11.128095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:11.128168 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:11.128286 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:11.129903 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.130635 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.132100 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.132534 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.132647 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:11.135964 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:11.135968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:11.135971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:11.135998 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:11.139109 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.139990 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.139995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.139998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.140606 (+ 608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.141233 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.141238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.141242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.141571 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:11.141729 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:11.142548 (+ 819us) server_negotiation.cc:299] Negotiation successful
0620 15:16:11.142773 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 15:16:11.143972 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:11.124553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34467 (local address 127.0.97.67:54525)
0620 15:16:11.125359 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:11.125416 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:11.127839 (+ 2423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:11.128592 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:11.128600 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:11.129220 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:11.129770 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:11.129787 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.130793 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.130801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:11.131981 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:11.131992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.132764 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.132773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:11.133073 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:11.133754 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:11.133775 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:11.135773 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:11.139262 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:11.139271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:11.139286 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:11.139857 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:11.140747 (+ 890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:11.140752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:11.140755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:11.140982 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:11.141727 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:11.141735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:11.142146 (+ 411us) client_negotiation.cc:769] Sending connection context
0620 15:16:11.142457 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 15:16:11.142911 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":267,"threads_started":1}
I20260620 15:16:11.145144 897 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34467
I20260620 15:16:11.145421 897 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:11.146072 897 heartbeater.cc:511] Master 127.0.97.126:34467 requested a full tablet report, sending...
I20260620 15:16:11.147338 441 ts_manager.cc:194] Registered new tserver with Master: 63c84f438b1a43c388ab01f1b13d3c69 (127.0.97.67:39645)
I20260620 15:16:11.148026 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:54525
I20260620 15:16:11.153401 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968571, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260620 15:16:11.183308 630 heartbeater.cc:503] Master 127.0.97.126:34467 was elected leader, sending a full tablet report...
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968571, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:11.196275 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:11.180470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.0.97.65:36961)
0620 15:16:11.181117 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:11.181124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:11.181264 (+ 140us) server_negotiation.cc:407] Connection header received
0620 15:16:11.181458 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:11.181466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:11.181672 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:11.181811 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:11.183248 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.184525 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.186053 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.186616 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.186725 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:11.190905 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:11.190914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:11.190926 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:11.190954 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:11.193587 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.194308 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.194315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.194318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.194428 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.194798 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.194805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.194809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.195105 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:11.195437 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:11.195704 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 15:16:11.195944 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":336,"threads_started":1}
W20260620 15:16:11.197494 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43802
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:11.235854 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:11.223088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.0.97.65:36961)
0620 15:16:11.223316 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:11.223322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:11.223545 (+ 223us) server_negotiation.cc:407] Connection header received
0620 15:16:11.223753 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:11.223757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:11.223860 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:11.223957 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:11.225257 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.225951 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.227122 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.227562 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.227677 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:11.231594 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:11.231599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:11.231602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:11.231629 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:11.233971 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.234517 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.234520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.234522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.234621 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.234914 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.234917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.234919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.235121 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:11.235290 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:11.235496 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 15:16:11.235633 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:16:11.237234 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 502
I20260620 15:16:11.248780 903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:11.238205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.0.97.65:36961)
0620 15:16:11.238404 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:11.238411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:11.238614 (+ 203us) server_negotiation.cc:407] Connection header received
0620 15:16:11.238799 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:11.238804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:11.238858 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:11.238946 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:11.240003 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.240693 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.241850 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.242242 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.242364 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:11.244329 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:11.244333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:11.244336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:11.244361 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:11.246572 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.247118 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.247124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.247127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.247248 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.247563 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.247569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.247572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.247777 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:11.247980 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:11.248270 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 15:16:11.248450 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:16:11.249464 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:11.332193 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 502
I20260620 15:16:11.348075 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 636
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:16:11.363979 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:11.348962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60244 (local address 127.0.97.66:34075)
0620 15:16:11.349478 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:11.349485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:11.349509 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:16:11.349577 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:11.349583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:11.349805 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:11.349986 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:11.351205 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.352257 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.353607 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.354275 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.354428 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:11.358609 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:11.358651 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:11.358664 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:11.358696 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:11.361630 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.362253 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.362265 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.362267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.362378 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.362742 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.362745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.362747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.363043 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:11.363264 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:11.363474 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 15:16:11.363656 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":258,"threads_started":1}
I20260620 15:16:11.365294 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:11.446524 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 636
I20260620 15:16:11.469062 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 769
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:16:11.485834 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:11.469969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.0.97.67:39645)
0620 15:16:11.470541 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:11.470548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:11.470577 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:16:11.470635 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:11.470641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:11.470856 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:11.470988 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:11.472254 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.473330 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.474620 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:11.475150 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:11.475240 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:11.479456 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:11.479467 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:11.479481 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:11.479514 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:11.483171 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.483852 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.483859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.483861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.483972 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:11.484332 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:11.484336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:11.484338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:11.484764 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:11.484980 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:11.485252 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:16:11.485436 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":294,"threads_started":1}
I20260620 15:16:11.487072 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:11.568033 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 769
I20260620 15:16:11.584028 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 409
I20260620 15:16:11.585381 471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:11.684754 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 409
2026-06-20T15:16:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5220 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[918](info): set up 2 sockets
Jun 20 15:16:11 dist-test-slave-7s64 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260620 15:16:12.752447 389 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:12 dist-test-slave-7s64 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:16:12.912230 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38875
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:13.217795 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:13.218048 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:13.218132 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:13.225042 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:13.225131 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:13.225175 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:13.225229 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:13.225271 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:13.234211 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:13.235265 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:13.236378 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:13.247428 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:13.247433 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:13.247654 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:13.248317 931 server_base.cc:1034] running on GCE node
I20260620 15:16:13.249082 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:13.250290 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:13.251480 931 hybrid_clock.cc:648] HybridClock initialized: now 1781968573251438 us; error 29 us; skew 500 ppm
I20260620 15:16:13.251811 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:13 dist-test-slave-7s64 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:13.260418 931 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:16:13.261938 931 webserver.cc:460] Webserver started at http://127.0.97.126:40397/ using document root <none> and password file <none>
I20260620 15:16:13.262530 931 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:13.262620 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:13.262851 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:13.265573 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "2955964c1e1f473a939b1f827f31ffa5"
format_stamp: "Formatted at 2026-06-20 15:16:13 on dist-test-slave-7s64"
server_key: "a1dd1e6e828885d5836c4abe9a15948f"
server_key_iv: "6fb1ba07ef58d85540467c6301854126"
server_key_version: "encryptionkey@0"
I20260620 15:16:13.266211 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "2955964c1e1f473a939b1f827f31ffa5"
format_stamp: "Formatted at 2026-06-20 15:16:13 on dist-test-slave-7s64"
server_key: "a1dd1e6e828885d5836c4abe9a15948f"
server_key_iv: "6fb1ba07ef58d85540467c6301854126"
server_key_version: "encryptionkey@0"
I20260620 15:16:13.270823 931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:16:13.273988 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:13.275246 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:16:13.275434 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "2955964c1e1f473a939b1f827f31ffa5"
format_stamp: "Formatted at 2026-06-20 15:16:13 on dist-test-slave-7s64"
server_key: "a1dd1e6e828885d5836c4abe9a15948f"
server_key_iv: "6fb1ba07ef58d85540467c6301854126"
server_key_version: "encryptionkey@0"
I20260620 15:16:13.275625 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:13.290848 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:13.294965 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:13.295233 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:13.312932 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:38875
I20260620 15:16:13.312968 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:38875 every 8 connection(s)
I20260620 15:16:13.314256 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:13.315079 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 931
I20260620 15:16:13.315325 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:13.315989 389 external_mini_cluster.cc:1428] Setting key 8bf73444a8a2afffa9466094b03fbea5
I20260620 15:16:13.318624 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:16:13 dist-test-slave-7s64 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:13.331081 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5: Bootstrap starting.
I20260620 15:16:13.334883 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:13.336120 998 log.cc:824] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:13.339219 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5: No bootstrap required, opened a new log
I20260620 15:16:13.340149 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:13.317932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36870 (local address 127.0.97.126:38875)
0620 15:16:13.318722 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:13.318738 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:13.318784 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:16:13.321098 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:13.321123 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:13.321606 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:13.322030 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:13.323285 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:13.325328 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:13.326513 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:13.327512 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:13.327715 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:13.331991 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:13.332002 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:13.332018 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:13.332057 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:13.335896 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:13.336578 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:13.336591 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:13.336602 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:13.336819 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:13.337214 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:13.337217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:13.337219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:13.337836 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:13.338102 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:13.338481 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 15:16:13.338869 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":311,"threads_started":1}
I20260620 15:16:13.347110 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2955964c1e1f473a939b1f827f31ffa5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38875 } }
I20260620 15:16:13.347412 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:13.347481 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2955964c1e1f473a939b1f827f31ffa5, State: Initialized, Role: FOLLOWER
I20260620 15:16:13.347983 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2955964c1e1f473a939b1f827f31ffa5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38875 } }
I20260620 15:16:13.348165 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:13.348299 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:13.348500 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:13.350562 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2955964c1e1f473a939b1f827f31ffa5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38875 } }
I20260620 15:16:13.350976 998 leader_election.cc:302] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2955964c1e1f473a939b1f827f31ffa5; no voters:
I20260620 15:16:13.351473 998 leader_election.cc:288] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:13.351608 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:13.352227 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [term 1 LEADER]: Becoming Leader. State: Replica: 2955964c1e1f473a939b1f827f31ffa5, State: Running, Role: LEADER
I20260620 15:16:13.352711 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2955964c1e1f473a939b1f827f31ffa5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38875 } }
I20260620 15:16:13.353412 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:13.355198 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2955964c1e1f473a939b1f827f31ffa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2955964c1e1f473a939b1f827f31ffa5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38875 } } }
I20260620 15:16:13.355576 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:13.355434 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2955964c1e1f473a939b1f827f31ffa5. Latest consensus state: current_term: 1 leader_uuid: "2955964c1e1f473a939b1f827f31ffa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2955964c1e1f473a939b1f827f31ffa5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38875 } } }
I20260620 15:16:13.355695 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:13.356575 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:13.361824 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:13.369943 1008 catalog_manager.cc:1347] Generated new cluster ID: 60e6cec90dd84255879b77fb6363ef27
I20260620 15:16:13.370041 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:13.384315 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:13.386049 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:13.397274 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2955964c1e1f473a939b1f827f31ffa5: Generated new TSK 0
I20260620 15:16:13.398223 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:16:13.452482 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38875
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:13.765591 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:13.765969 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:13.766139 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:13.773092 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:13.773222 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:13.773411 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:13.783565 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38875
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:13.784755 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:13.786013 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:13.798730 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:13.798761 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:13.798777 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:13.799628 1024 server_base.cc:1034] running on GCE node
I20260620 15:16:13.800163 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:13.801071 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:13.802294 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781968573802255 us; error 32 us; skew 500 ppm
I20260620 15:16:13.802614 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:13 dist-test-slave-7s64 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:13.805802 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:16:13.807132 1024 webserver.cc:460] Webserver started at http://127.0.97.65:42197/ using document root <none> and password file <none>
I20260620 15:16:13.807962 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:13.808116 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:13.808413 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:13.811131 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "42ef52bf9e874f2c8fe5acb57fea7a08"
format_stamp: "Formatted at 2026-06-20 15:16:13 on dist-test-slave-7s64"
server_key: "5bbf96c25bf1cf238548f6fbef5d4534"
server_key_iv: "b7e1e39735c503c1d189372df7cd66d8"
server_key_version: "encryptionkey@0"
I20260620 15:16:13.811872 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "42ef52bf9e874f2c8fe5acb57fea7a08"
format_stamp: "Formatted at 2026-06-20 15:16:13 on dist-test-slave-7s64"
server_key: "5bbf96c25bf1cf238548f6fbef5d4534"
server_key_iv: "b7e1e39735c503c1d189372df7cd66d8"
server_key_version: "encryptionkey@0"
I20260620 15:16:13.816784 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:16:13.819952 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:13.821223 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:13.821553 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "42ef52bf9e874f2c8fe5acb57fea7a08"
format_stamp: "Formatted at 2026-06-20 15:16:13 on dist-test-slave-7s64"
server_key: "5bbf96c25bf1cf238548f6fbef5d4534"
server_key_iv: "b7e1e39735c503c1d189372df7cd66d8"
server_key_version: "encryptionkey@0"
I20260620 15:16:13.821753 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:13.841149 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:13.845245 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:13.845510 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:13.846385 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:13.847587 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:13.847658 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:13.847826 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:13.847882 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:13.880952 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:45217
I20260620 15:16:13.880998 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:45217 every 8 connection(s)
I20260620 15:16:13.882182 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:13.888113 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 1024
I20260620 15:16:13.888330 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:13.889135 389 external_mini_cluster.cc:1428] Setting key 7195bce871dbe509af62dcd1c5776f1e
Jun 20 15:16:13 dist-test-slave-7s64 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:13.905406 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:13.884751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:46735 (local address 127.0.97.126:38875)
0620 15:16:13.885212 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:13.885219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:13.886210 (+ 991us) server_negotiation.cc:407] Connection header received
0620 15:16:13.888073 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:13.888078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:13.888140 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:13.888287 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:13.890261 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:13.891308 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:13.894020 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:13.894409 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:13.894510 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:13.898432 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:13.898441 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:13.898449 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:13.898495 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:13.902065 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:13.902870 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:13.902877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:13.902880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:13.903027 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:13.903431 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:13.903440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:13.903446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:13.903843 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:13.904011 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:13.904733 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 15:16:13.904971 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":207,"threads_started":1}
I20260620 15:16:13.906150 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:13.885385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38875 (local address 127.0.97.65:46735)
0620 15:16:13.886052 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:13.886094 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:13.887765 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:13.888585 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:13.888598 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:13.889357 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:13.890079 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:13.890102 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:13.891463 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:13.891470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:13.893854 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:13.893874 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:13.894633 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:13.894642 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:13.894971 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:13.895720 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:13.895763 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:13.898180 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:13.902208 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:13.902214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:13.902227 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:13.902727 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:13.903132 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:13.903136 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:13.903138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:13.903306 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:13.904004 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:13.904010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:13.904308 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 15:16:13.904612 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 15:16:13.904986 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":267,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:16:13.907871 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38875
I20260620 15:16:13.908123 1152 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:13.908824 1152 heartbeater.cc:511] Master 127.0.97.126:38875 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260620 15:16:13.911119 963 ts_manager.cc:194] Registered new tserver with Master: 42ef52bf9e874f2c8fe5acb57fea7a08 (127.0.97.65:45217)
I20260620 15:16:13.912539 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:46735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:16:13.931424 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38875
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:14.240276 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:14.240558 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:14.240670 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:14.247313 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:14.247399 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:14.247503 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:14.256474 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38875
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:14.257541 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:14.258808 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:14.270994 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:14.271107 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:14.271139 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:14.272027 1158 server_base.cc:1034] running on GCE node
I20260620 15:16:14.272634 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:14.273490 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:14.274706 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781968574274675 us; error 42 us; skew 500 ppm
I20260620 15:16:14.275003 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:14 dist-test-slave-7s64 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:14.277907 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:16:14.279347 1158 webserver.cc:460] Webserver started at http://127.0.97.66:46865/ using document root <none> and password file <none>
I20260620 15:16:14.280112 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:14.280292 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:14.280676 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:14.283460 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "c095f0342871416eb6774244b877ba4c"
format_stamp: "Formatted at 2026-06-20 15:16:14 on dist-test-slave-7s64"
server_key: "8f0c03914b6d94ce5336c3770b52bcb9"
server_key_iv: "07cf6f544611ce5c7d636fa28084619e"
server_key_version: "encryptionkey@0"
I20260620 15:16:14.284276 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "c095f0342871416eb6774244b877ba4c"
format_stamp: "Formatted at 2026-06-20 15:16:14 on dist-test-slave-7s64"
server_key: "8f0c03914b6d94ce5336c3770b52bcb9"
server_key_iv: "07cf6f544611ce5c7d636fa28084619e"
server_key_version: "encryptionkey@0"
I20260620 15:16:14.289264 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 15:16:14.292454 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:14.293727 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:16:14.293886 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "c095f0342871416eb6774244b877ba4c"
format_stamp: "Formatted at 2026-06-20 15:16:14 on dist-test-slave-7s64"
server_key: "8f0c03914b6d94ce5336c3770b52bcb9"
server_key_iv: "07cf6f544611ce5c7d636fa28084619e"
server_key_version: "encryptionkey@0"
I20260620 15:16:14.294070 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:14.315637 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:14.319916 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:14.320182 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:14.321141 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:14.322451 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:14.322556 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:14.322696 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:14.322767 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:14.355389 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:44327
I20260620 15:16:14.355468 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:44327 every 8 connection(s)
I20260620 15:16:14.356714 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:14.365388 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 1158
I20260620 15:16:14.365566 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:14.366039 389 external_mini_cluster.cc:1428] Setting key a52629bb6147bee4791ce95d21789693
Jun 20 15:16:14 dist-test-slave-7s64 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:14.376438 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.359301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56347 (local address 127.0.97.126:38875)
0620 15:16:14.359489 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:14.359493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:14.360776 (+ 1283us) server_negotiation.cc:407] Connection header received
0620 15:16:14.362581 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:14.362587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:14.362679 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:14.362787 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:14.364270 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.365049 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.366792 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.367283 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.367381 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.370882 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:14.370887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:14.370890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:14.370917 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:14.373457 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:14.374327 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:14.374331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:14.374333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:14.374455 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:14.374907 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:14.374910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:14.374912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:14.375158 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:14.375264 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:14.376052 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 15:16:14.376202 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
I20260620 15:16:14.377909 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.359755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38875 (local address 127.0.97.66:56347)
0620 15:16:14.360623 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:14.360660 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:14.362321 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:14.363007 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:14.363016 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:14.363525 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:14.364104 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.364124 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.365195 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.365201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.366619 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.366633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.367496 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.367504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.367773 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.368643 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:14.368663 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:14.370657 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:14.373591 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:14.373602 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:14.373620 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:14.374203 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:14.374573 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:14.374581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:14.374587 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:14.374806 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:14.375279 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:14.375289 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:14.375698 (+ 409us) client_negotiation.cc:769] Sending connection context
0620 15:16:14.375982 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 15:16:14.376533 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":354,"threads_started":1}
I20260620 15:16:14.379469 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38875
I20260620 15:16:14.379803 1286 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:14.380715 1286 heartbeater.cc:511] Master 127.0.97.126:38875 requested a full tablet report, sending...
I20260620 15:16:14.382258 963 ts_manager.cc:194] Registered new tserver with Master: c095f0342871416eb6774244b877ba4c (127.0.97.66:44327)
I20260620 15:16:14.383035 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:56347
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:16:14.407349 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38875
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:14.715132 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:14.715418 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:14.715590 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:14.722628 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:14.722772 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:14.722934 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:14.732187 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38875
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:14.733405 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:14.734576 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:14.747509 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:14.747535 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:14.747617 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:14.748610 1291 server_base.cc:1034] running on GCE node
I20260620 15:16:14.749150 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:14.749980 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:14.751207 1291 hybrid_clock.cc:648] HybridClock initialized: now 1781968574751168 us; error 51 us; skew 500 ppm
I20260620 15:16:14.751492 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:14 dist-test-slave-7s64 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:14.754541 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:16:14.755785 1291 webserver.cc:460] Webserver started at http://127.0.97.67:43555/ using document root <none> and password file <none>
I20260620 15:16:14.756419 1291 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:14.756577 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:14.756866 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:14.759677 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "3765f19cb713435c87331d0dfc09b207"
format_stamp: "Formatted at 2026-06-20 15:16:14 on dist-test-slave-7s64"
server_key: "136168a0064313b4f07836c6596793bf"
server_key_iv: "f069125c0f6a03027e57f53888703e8b"
server_key_version: "encryptionkey@0"
I20260620 15:16:14.760464 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "3765f19cb713435c87331d0dfc09b207"
format_stamp: "Formatted at 2026-06-20 15:16:14 on dist-test-slave-7s64"
server_key: "136168a0064313b4f07836c6596793bf"
server_key_iv: "f069125c0f6a03027e57f53888703e8b"
server_key_version: "encryptionkey@0"
I20260620 15:16:14.765259 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:16:14.768371 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:14.769554 1291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:16:14.769718 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "3765f19cb713435c87331d0dfc09b207"
format_stamp: "Formatted at 2026-06-20 15:16:14 on dist-test-slave-7s64"
server_key: "136168a0064313b4f07836c6596793bf"
server_key_iv: "f069125c0f6a03027e57f53888703e8b"
server_key_version: "encryptionkey@0"
I20260620 15:16:14.769898 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:14.794737 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:14.798954 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:14.799232 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:14.800226 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:14.801491 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:14.801587 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:14.801728 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:14.801798 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:14.833539 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:41461
I20260620 15:16:14.833633 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:41461 every 8 connection(s)
I20260620 15:16:14.834856 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:14.845489 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 1291
I20260620 15:16:14.845667 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:14.846240 389 external_mini_cluster.cc:1428] Setting key 394b428a2c69399eda521cec734db995
Jun 20 15:16:14 dist-test-slave-7s64 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:14.853927 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.837354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:50325 (local address 127.0.97.126:38875)
0620 15:16:14.837694 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:14.837699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:14.838600 (+ 901us) server_negotiation.cc:407] Connection header received
0620 15:16:14.840362 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:14.840367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:14.840453 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:14.840584 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:14.841963 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.842684 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.844161 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.844618 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.844751 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.848326 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:14.848331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:14.848334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:14.848364 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:14.850972 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:14.851664 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:14.851670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:14.851674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:14.851797 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:14.852240 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:14.852247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:14.852250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:14.852556 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:14.852692 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:14.853354 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 15:16:14.853547 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260620 15:16:14.854805 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.837781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38875 (local address 127.0.97.67:50325)
0620 15:16:14.838443 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:14.838484 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:14.840099 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:14.840805 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:14.840814 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:14.841288 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:14.841805 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.841820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.842831 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.842838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.844016 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.844030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.844824 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.844833 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.845072 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.845748 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:14.845778 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:14.848129 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:14.851104 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:14.851113 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:14.851126 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:14.851546 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:14.851929 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:14.851934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:14.851937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:14.852126 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:14.852679 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:14.852686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:14.852984 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 15:16:14.853266 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 15:16:14.853767 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":270,"threads_started":1}
I20260620 15:16:14.856015 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38875
I20260620 15:16:14.856288 1419 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:14.856979 1419 heartbeater.cc:511] Master 127.0.97.126:38875 requested a full tablet report, sending...
I20260620 15:16:14.858317 963 ts_manager.cc:194] Registered new tserver with Master: 3765f19cb713435c87331d0dfc09b207 (127.0.97.67:41461)
I20260620 15:16:14.859143 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:50325
I20260620 15:16:14.861274 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:16:14.879171 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.867239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.0.97.126:38875)
0620 15:16:14.867404 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:14.867408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:14.867885 (+ 477us) server_negotiation.cc:407] Connection header received
0620 15:16:14.867999 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:14.868002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:14.868057 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:14.868140 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:14.869565 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.870668 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.871839 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.872201 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.872343 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.874471 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:14.874476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:14.874479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:14.874509 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:14.877014 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:14.877543 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:14.877547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:14.877549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:14.877673 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:14.877962 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:14.877965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:14.877967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:14.878219 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:14.878472 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:14.878829 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 15:16:14.878967 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:16:14.885253 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:16:14.892031 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:14.916106 1152 heartbeater.cc:503] Master 127.0.97.126:38875 was elected leader, sending a full tablet report...
I20260620 15:16:14.929977 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.904005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.0.97.67:41461)
0620 15:16:14.921033 (+ 17028us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:14.921045 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:14.921091 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:16:14.921234 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:14.921246 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:14.921596 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:14.921856 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:14.922932 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.924970 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.927364 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.928858 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.928986 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.929221 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:14.929490 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 15:16:14.929651 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16843,"thread_start_us":16710,"threads_started":1}
I20260620 15:16:14.930097 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.903485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:41461 (local address 127.0.0.1:42426)
0620 15:16:14.920504 (+ 17019us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:14.920537 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:14.920691 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:14.922118 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:14.922129 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:14.922212 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:14.922697 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.922711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.925155 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.925160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.927199 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.927210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.929040 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.929046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.929241 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.929298 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 15:16:14.929434 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 15:16:14.929645 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16832,"mutex_wait_us":47,"thread_start_us":18869,"threads_started":1}
I20260620 15:16:14.930960 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.903190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44327 (local address 127.0.0.1:38282)
0620 15:16:14.904060 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:14.904108 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:14.904295 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:14.923556 (+ 19261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:14.923560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:14.923581 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:14.923916 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.923928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.925718 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.925724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.927970 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.927985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.929707 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.929712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.929879 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.929907 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 15:16:14.929977 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 15:16:14.930117 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":21,"thread_start_us":494,"threads_started":1}
I20260620 15:16:14.931717 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.904926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.0.97.65:45217)
0620 15:16:14.920148 (+ 15222us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:14.920161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:14.922847 (+ 2686us) server_negotiation.cc:407] Connection header received
0620 15:16:14.923507 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:14.923516 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:14.923751 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:14.923943 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:14.924563 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.926145 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.928665 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.930056 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.930178 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.930383 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:14.931222 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 15:16:14.931385 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15009,"thread_start_us":15241,"threads_started":1}
I20260620 15:16:14.932397 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.904652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:45217 (local address 127.0.0.1:57698)
0620 15:16:14.922731 (+ 18079us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:14.922760 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:14.923336 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:14.923953 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:14.923958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:14.923985 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:14.924356 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.924368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.926280 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.926285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.928433 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.928445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.930696 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.930705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.930945 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.930979 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:16:14.931912 (+ 933us) client_negotiation.cc:240] Negotiation successful
0620 15:16:14.932041 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17909,"thread_start_us":16148,"threads_started":1}
I20260620 15:16:14.933775 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.903364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.0.97.66:44327)
0620 15:16:14.922921 (+ 19557us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:14.922928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:14.922954 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:16:14.923050 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:14.923059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:14.923363 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:14.923579 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:14.924127 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.925576 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.928151 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.929495 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.930989 (+ 1494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.931380 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:14.931439 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 15:16:14.931597 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19395,"thread_start_us":19303,"threads_started":1}
I20260620 15:16:14.934916 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 8311fad1f6544f5cbf1deb458c82e8da (DEFAULT_TABLE table=test-table [id=0b4ccb5b9e5343f9b305ed7627ce2405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:14.936437 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8311fad1f6544f5cbf1deb458c82e8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:14.936056 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 8311fad1f6544f5cbf1deb458c82e8da (DEFAULT_TABLE table=test-table [id=0b4ccb5b9e5343f9b305ed7627ce2405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:14.936372 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 8311fad1f6544f5cbf1deb458c82e8da (DEFAULT_TABLE table=test-table [id=0b4ccb5b9e5343f9b305ed7627ce2405]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:14.937601 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8311fad1f6544f5cbf1deb458c82e8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:14.937692 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8311fad1f6544f5cbf1deb458c82e8da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:14.947896 1437 tablet_bootstrap.cc:492] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c: Bootstrap starting.
I20260620 15:16:14.951634 1437 tablet_bootstrap.cc:654] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:14.952692 1437 log.cc:824] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:14.953254 1439 tablet_bootstrap.cc:492] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08: Bootstrap starting.
I20260620 15:16:14.956619 1438 tablet_bootstrap.cc:492] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207: Bootstrap starting.
I20260620 15:16:14.957623 1439 tablet_bootstrap.cc:654] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:14.959196 1439 log.cc:824] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:14.960614 1438 tablet_bootstrap.cc:654] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:14.961760 1438 log.cc:824] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:14.963160 1439 tablet_bootstrap.cc:492] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08: No bootstrap required, opened a new log
I20260620 15:16:14.963419 1439 ts_tablet_manager.cc:1397] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260620 15:16:14.964814 1437 tablet_bootstrap.cc:492] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c: No bootstrap required, opened a new log
I20260620 15:16:14.964933 1438 tablet_bootstrap.cc:492] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207: No bootstrap required, opened a new log
I20260620 15:16:14.965075 1437 ts_tablet_manager.cc:1397] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.004s
I20260620 15:16:14.965193 1438 ts_tablet_manager.cc:1397] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20260620 15:16:14.972241 1438 raft_consensus.cc:348] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c095f0342871416eb6774244b877ba4c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44327 } } peers { permanent_uuid: "3765f19cb713435c87331d0dfc09b207" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41461 } } peers { permanent_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45217 } }
I20260620 15:16:14.972608 1438 raft_consensus.cc:374] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:14.972692 1438 raft_consensus.cc:729] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3765f19cb713435c87331d0dfc09b207, State: Initialized, Role: FOLLOWER
I20260620 15:16:14.973310 1438 consensus_queue.cc:260] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c095f0342871416eb6774244b877ba4c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44327 } } peers { permanent_uuid: "3765f19cb713435c87331d0dfc09b207" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41461 } } peers { permanent_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45217 } }
I20260620 15:16:14.974184 1439 raft_consensus.cc:348] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c095f0342871416eb6774244b877ba4c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44327 } } peers { permanent_uuid: "3765f19cb713435c87331d0dfc09b207" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41461 } } peers { permanent_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45217 } }
I20260620 15:16:14.974589 1439 raft_consensus.cc:374] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:14.974679 1439 raft_consensus.cc:729] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ef52bf9e874f2c8fe5acb57fea7a08, State: Initialized, Role: FOLLOWER
I20260620 15:16:14.974907 1419 heartbeater.cc:503] Master 127.0.97.126:38875 was elected leader, sending a full tablet report...
I20260620 15:16:14.975479 1438 ts_tablet_manager.cc:1428] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260620 15:16:14.975342 1439 consensus_queue.cc:260] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c095f0342871416eb6774244b877ba4c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44327 } } peers { permanent_uuid: "3765f19cb713435c87331d0dfc09b207" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41461 } } peers { permanent_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45217 } }
I20260620 15:16:14.975873 1437 raft_consensus.cc:348] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c095f0342871416eb6774244b877ba4c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44327 } } peers { permanent_uuid: "3765f19cb713435c87331d0dfc09b207" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41461 } } peers { permanent_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45217 } }
I20260620 15:16:14.976275 1437 raft_consensus.cc:374] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:14.976362 1437 raft_consensus.cc:729] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c095f0342871416eb6774244b877ba4c, State: Initialized, Role: FOLLOWER
I20260620 15:16:14.976974 1439 ts_tablet_manager.cc:1428] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260620 15:16:14.977025 1437 consensus_queue.cc:260] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c095f0342871416eb6774244b877ba4c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44327 } } peers { permanent_uuid: "3765f19cb713435c87331d0dfc09b207" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41461 } } peers { permanent_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45217 } }
I20260620 15:16:14.979506 1286 heartbeater.cc:503] Master 127.0.97.126:38875 was elected leader, sending a full tablet report...
I20260620 15:16:14.981406 1437 ts_tablet_manager.cc:1428] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260620 15:16:14.985587 1443 raft_consensus.cc:482] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:14.985839 1443 raft_consensus.cc:504] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c095f0342871416eb6774244b877ba4c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44327 } } peers { permanent_uuid: "3765f19cb713435c87331d0dfc09b207" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41461 } } peers { permanent_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45217 } }
I20260620 15:16:14.987282 1443 leader_election.cc:288] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c095f0342871416eb6774244b877ba4c (127.0.97.66:44327), 42ef52bf9e874f2c8fe5acb57fea7a08 (127.0.97.65:45217)
I20260620 15:16:14.993372 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.987713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:45965 (local address 127.0.97.66:44327)
0620 15:16:14.987928 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:14.987936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:14.987999 (+ 63us) server_negotiation.cc:407] Connection header received
0620 15:16:14.988238 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:14.988245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:14.988328 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:14.988449 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:14.989053 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.989774 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.991584 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.992389 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.992578 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.992678 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:14.992967 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 15:16:14.993094 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:16:14.993384 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.987702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44327 (local address 127.0.97.67:45965)
0620 15:16:14.987926 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:14.987947 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:14.988063 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:14.988691 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:14.988696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:14.988729 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:14.988910 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.988918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.989927 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.989935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.991376 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.991392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.992594 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.992600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.992796 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.992821 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:16:14.992931 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 15:16:14.993076 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 15:16:14.994086 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.988232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:47083 (local address 127.0.97.65:45217)
0620 15:16:14.988412 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:14.988420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:14.988874 (+ 454us) server_negotiation.cc:407] Connection header received
0620 15:16:14.989049 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:14.989054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:14.989118 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:14.989223 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:14.989681 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.990353 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.992364 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.993222 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.993371 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.993465 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:14.993749 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:16:14.993867 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:16:14.994233 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:14.988112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:45217 (local address 127.0.97.67:47083)
0620 15:16:14.988785 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:14.988798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:14.988935 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:14.989237 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:14.989241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:14.989264 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:14.989551 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.989559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.990495 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.990505 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.992230 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:14.992241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:14.993418 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:14.993424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:14.993584 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:14.993614 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:16:14.993730 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 15:16:14.993846 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":402,"threads_started":1}
I20260620 15:16:14.994513 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8311fad1f6544f5cbf1deb458c82e8da" candidate_uuid: "3765f19cb713435c87331d0dfc09b207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c095f0342871416eb6774244b877ba4c" is_pre_election: true
I20260620 15:16:14.995005 1241 raft_consensus.cc:2454] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3765f19cb713435c87331d0dfc09b207 in term 0.
I20260620 15:16:14.995601 1310 leader_election.cc:302] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3765f19cb713435c87331d0dfc09b207, c095f0342871416eb6774244b877ba4c; no voters:
I20260620 15:16:14.995922 1443 raft_consensus.cc:2781] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:14.995808 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8311fad1f6544f5cbf1deb458c82e8da" candidate_uuid: "3765f19cb713435c87331d0dfc09b207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" is_pre_election: true
I20260620 15:16:14.996037 1443 raft_consensus.cc:482] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:14.996145 1443 raft_consensus.cc:3037] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:14.996337 1107 raft_consensus.cc:2454] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3765f19cb713435c87331d0dfc09b207 in term 0.
I20260620 15:16:14.998890 1443 raft_consensus.cc:504] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c095f0342871416eb6774244b877ba4c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44327 } } peers { permanent_uuid: "3765f19cb713435c87331d0dfc09b207" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41461 } } peers { permanent_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45217 } }
I20260620 15:16:14.999507 1443 leader_election.cc:288] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [CANDIDATE]: Term 1 election: Requested vote from peers c095f0342871416eb6774244b877ba4c (127.0.97.66:44327), 42ef52bf9e874f2c8fe5acb57fea7a08 (127.0.97.65:45217)
I20260620 15:16:14.999935 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8311fad1f6544f5cbf1deb458c82e8da" candidate_uuid: "3765f19cb713435c87331d0dfc09b207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c095f0342871416eb6774244b877ba4c"
I20260620 15:16:14.999992 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8311fad1f6544f5cbf1deb458c82e8da" candidate_uuid: "3765f19cb713435c87331d0dfc09b207" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08"
I20260620 15:16:15.000109 1241 raft_consensus.cc:3037] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:15.000166 1107 raft_consensus.cc:3037] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:15.002214 1107 raft_consensus.cc:2454] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3765f19cb713435c87331d0dfc09b207 in term 1.
I20260620 15:16:15.002261 1241 raft_consensus.cc:2454] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3765f19cb713435c87331d0dfc09b207 in term 1.
I20260620 15:16:15.002573 1310 leader_election.cc:302] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3765f19cb713435c87331d0dfc09b207, 42ef52bf9e874f2c8fe5acb57fea7a08; no voters:
I20260620 15:16:15.002851 1443 raft_consensus.cc:2781] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:15.003228 1443 raft_consensus.cc:686] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [term 1 LEADER]: Becoming Leader. State: Replica: 3765f19cb713435c87331d0dfc09b207, State: Running, Role: LEADER
I20260620 15:16:15.003650 1443 consensus_queue.cc:237] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c095f0342871416eb6774244b877ba4c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44327 } } peers { permanent_uuid: "3765f19cb713435c87331d0dfc09b207" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41461 } } peers { permanent_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45217 } }
I20260620 15:16:15.007969 961 catalog_manager.cc:5495] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3765f19cb713435c87331d0dfc09b207 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "3765f19cb713435c87331d0dfc09b207" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c095f0342871416eb6774244b877ba4c" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3765f19cb713435c87331d0dfc09b207" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42ef52bf9e874f2c8fe5acb57fea7a08" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45217 } health_report { overall_health: UNKNOWN } } }
Jun 20 15:16:15 dist-test-slave-7s64 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:15.034857 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:15.020139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.0.97.65:45217)
0620 15:16:15.020329 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:15.020336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:15.020985 (+ 649us) server_negotiation.cc:407] Connection header received
0620 15:16:15.021112 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:15.021116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:15.021204 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:15.021293 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:15.022694 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.023426 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.024688 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.025163 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.025260 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:15.029729 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:15.029741 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:15.029755 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:15.029788 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:15.032272 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.032935 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.032942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.032944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.033053 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.033447 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.033457 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.033459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.033791 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:15.034084 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:15.034350 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 15:16:15.034550 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 15:16:15 dist-test-slave-7s64 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:16:15 dist-test-slave-7s64 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968575, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:15.079067 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:15.065641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.0.97.65:45217)
0620 15:16:15.065867 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:15.065876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:15.066235 (+ 359us) server_negotiation.cc:407] Connection header received
0620 15:16:15.066446 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:15.066450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:15.066510 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:15.066609 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:15.067857 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.068632 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.069879 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.070287 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.070413 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:15.074527 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:15.074532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:15.074535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:15.074565 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:15.076936 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.077427 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.077431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.077433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.077549 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.077893 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.077896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.077898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.078118 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:15.078380 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:15.078688 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 15:16:15.078842 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 15:16:15.086367 1420 tablet.cc:2369] T 8311fad1f6544f5cbf1deb458c82e8da P 3765f19cb713435c87331d0dfc09b207: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 15:16:15 dist-test-slave-7s64 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260620 15:16:15.108574 1287 tablet.cc:2369] T 8311fad1f6544f5cbf1deb458c82e8da P c095f0342871416eb6774244b877ba4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 15:16:15 dist-test-slave-7s64 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:15.118964 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:15.105928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.0.97.65:45217)
0620 15:16:15.106162 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:15.106171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:15.106404 (+ 233us) server_negotiation.cc:407] Connection header received
0620 15:16:15.106601 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:15.106605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:15.106677 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:15.106784 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:15.108007 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.108732 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.109880 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.110285 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.110406 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:15.114092 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:15.114097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:15.114100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:15.114128 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:15.116387 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.117021 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.117025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.117027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.117136 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.117515 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.117518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.117520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.117779 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:15.118399 (+ 620us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:15.118481 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 15:16:15.118732 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 15:16:15.120718 1067 tablet_service.cc:2908] Rejecting scan request for tablet 8311fad1f6544f5cbf1deb458c82e8da: Uninitialized: clean time has not yet been initialized
I20260620 15:16:15.123519 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 1024
W20260620 15:16:15.134495 1153 tablet.cc:2369] T 8311fad1f6544f5cbf1deb458c82e8da P 42ef52bf9e874f2c8fe5acb57fea7a08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:16:15.137647 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:15.124870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.0.97.65:45217)
0620 15:16:15.125078 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:15.125087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:15.125364 (+ 277us) server_negotiation.cc:407] Connection header received
0620 15:16:15.125477 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:15.125481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:15.125541 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:15.125613 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:15.126885 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.127556 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.128878 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.129287 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.129395 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:15.131445 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:15.131449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:15.131452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:15.131486 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:15.134704 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.135602 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.135608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.135611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.135727 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.136085 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.136092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.136095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.136362 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:15.136666 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:15.136954 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 15:16:15.137147 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:16:15.138671 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:15.228700 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 1024
I20260620 15:16:15.245814 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 1158
Jun 20 15:16:15 dist-test-slave-7s64 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:16:15.259959 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:15.246702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.0.97.66:44327)
0620 15:16:15.246937 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:15.246943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:15.246963 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:16:15.247114 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:15.247120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:15.247210 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:15.247319 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:15.248436 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.249237 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.250462 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.250867 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.250997 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:15.254802 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:15.254815 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:15.254829 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:15.254859 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:15.257604 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.258283 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.258291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.258293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.258405 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.258704 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.258707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.258709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.259092 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:15.259321 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:15.259515 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 15:16:15.259657 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 15:16:15.260699 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:15.345425 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 1158
I20260620 15:16:15.361662 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 1291
Jun 20 15:16:15 dist-test-slave-7s64 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:16:15.375984 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:15.362527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.0.97.67:41461)
0620 15:16:15.362685 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:15.362689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:15.362704 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:16:15.362757 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:15.362761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:15.362825 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:15.362905 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:15.364044 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.364812 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.365999 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:15.366382 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:15.366520 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:15.370383 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:15.370396 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:15.370411 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:15.370439 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:15.373255 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.373820 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.373829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.373833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.374029 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:15.374452 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:15.374456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:15.374458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:15.374826 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:15.375075 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:15.375352 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 15:16:15.375520 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:16:15.376757 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:15.465732 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 1291
I20260620 15:16:15.481942 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 931
I20260620 15:16:15.483076 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:15.588876 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 931
2026-06-20T15:16:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3908 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:15 dist-test-slave-7s64 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:15 dist-test-slave-7s64 krb5kdc[1466](info): set up 2 sockets
Jun 20 15:16:15 dist-test-slave-7s64 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260620 15:16:16.660362 389 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:16 dist-test-slave-7s64 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:16:16.822221 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37043
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:17.121901 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:17.122152 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:17.122282 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:17.129060 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:17.129143 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:16:17.129241 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:17.129345 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:17.129424 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:17.129513 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:17.139693 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37043
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:17.141419 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:17.143154 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:17.154542 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:17.154510 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:17.154469 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:17.155611 1479 server_base.cc:1034] running on GCE node
I20260620 15:16:17.156320 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:17.157622 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:17.158866 1479 hybrid_clock.cc:648] HybridClock initialized: now 1781968577158808 us; error 34 us; skew 500 ppm
I20260620 15:16:17.159205 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:17 dist-test-slave-7s64 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:17.162422 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:16:17.163869 1479 webserver.cc:460] Webserver started at http://127.0.97.126:46503/ using document root <none> and password file <none>
I20260620 15:16:17.164594 1479 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:17.164713 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:17.165031 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:17.167927 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "e407b154ee3748d9b9d7f35efef530f5"
format_stamp: "Formatted at 2026-06-20 15:16:17 on dist-test-slave-7s64"
server_key: "520887a380a69332f4448c4f0c4c418b"
server_key_iv: "10cdc6057b77139a3b0c7e7b0279113a"
server_key_version: "encryptionkey@0"
I20260620 15:16:17.168751 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "e407b154ee3748d9b9d7f35efef530f5"
format_stamp: "Formatted at 2026-06-20 15:16:17 on dist-test-slave-7s64"
server_key: "520887a380a69332f4448c4f0c4c418b"
server_key_iv: "10cdc6057b77139a3b0c7e7b0279113a"
server_key_version: "encryptionkey@0"
I20260620 15:16:17.173655 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:16:17.176999 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:17.178205 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:16:17.178381 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "e407b154ee3748d9b9d7f35efef530f5"
format_stamp: "Formatted at 2026-06-20 15:16:17 on dist-test-slave-7s64"
server_key: "520887a380a69332f4448c4f0c4c418b"
server_key_iv: "10cdc6057b77139a3b0c7e7b0279113a"
server_key_version: "encryptionkey@0"
I20260620 15:16:17.178586 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:17.193502 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:17.197603 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:17.197871 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:17.218179 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37043
I20260620 15:16:17.218185 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37043 every 8 connection(s)
I20260620 15:16:17.219488 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:17.222939 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:17.225679 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 1479
I20260620 15:16:17.225858 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:17.226632 389 external_mini_cluster.cc:1428] Setting key 7822ad89aa8cb918de6ea66526666ba1
I20260620 15:16:17.235142 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5: Bootstrap starting.
I20260620 15:16:17.238643 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:17.239890 1556 log.cc:824] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:16:17 dist-test-slave-7s64 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:17.242676 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5: No bootstrap required, opened a new log
I20260620 15:16:17.249006 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:17.228673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55216 (local address 127.0.97.126:37043)
0620 15:16:17.229454 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:17.229465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:17.229503 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:16:17.231229 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:17.231250 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:17.231626 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:17.231923 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:17.233655 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:17.234954 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:17.236685 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:17.237390 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:17.237525 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:17.242225 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:17.242233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:17.242246 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:17.242277 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:17.245338 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:17.245972 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:17.245978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:17.245983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:17.246121 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:17.246488 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:17.246492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:17.246494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:17.246977 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:17.247287 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:17.247642 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 15:16:17.248023 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":359,"threads_started":1}
I20260620 15:16:17.250154 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e407b154ee3748d9b9d7f35efef530f5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37043 } }
I20260620 15:16:17.250495 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:17.250607 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e407b154ee3748d9b9d7f35efef530f5, State: Initialized, Role: FOLLOWER
I20260620 15:16:17.251201 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e407b154ee3748d9b9d7f35efef530f5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37043 } }
I20260620 15:16:17.251452 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:17.251586 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:17.251727 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:17.253746 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e407b154ee3748d9b9d7f35efef530f5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37043 } }
I20260620 15:16:17.254142 1556 leader_election.cc:302] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e407b154ee3748d9b9d7f35efef530f5; no voters:
I20260620 15:16:17.254618 1556 leader_election.cc:288] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:17.254765 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:17.255439 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [term 1 LEADER]: Becoming Leader. State: Replica: e407b154ee3748d9b9d7f35efef530f5, State: Running, Role: LEADER
I20260620 15:16:17.255927 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e407b154ee3748d9b9d7f35efef530f5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37043 } }
I20260620 15:16:17.256733 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:17.258639 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e407b154ee3748d9b9d7f35efef530f5. Latest consensus state: current_term: 1 leader_uuid: "e407b154ee3748d9b9d7f35efef530f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e407b154ee3748d9b9d7f35efef530f5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37043 } } }
I20260620 15:16:17.258611 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e407b154ee3748d9b9d7f35efef530f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e407b154ee3748d9b9d7f35efef530f5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37043 } } }
I20260620 15:16:17.258999 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:17.258999 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:17.261305 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:17.266028 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:17.274519 1566 catalog_manager.cc:1347] Generated new cluster ID: 2c9fd9f893bd423d88a964d3d5b5a2a6
I20260620 15:16:17.274616 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:17.307731 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:17.309687 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:17.319557 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5: Generated new TSK 0
I20260620 15:16:17.320601 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:16:17.374330 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37043
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:17.697002 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:17.697260 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:17.697409 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:17.704463 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:17.704571 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:17.704707 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:16:17.704826 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:17.714051 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:37043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:17.715288 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:17.716598 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:17.729321 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:17.729481 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:17.729609 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:17.730520 1582 server_base.cc:1034] running on GCE node
I20260620 15:16:17.731072 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:17.731902 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:17.733131 1582 hybrid_clock.cc:648] HybridClock initialized: now 1781968577733068 us; error 28 us; skew 500 ppm
I20260620 15:16:17.733433 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:17 dist-test-slave-7s64 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:17.741086 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:16:17.743594 1582 webserver.cc:460] Webserver started at http://127.0.97.65:38285/ using document root <none> and password file <none>
I20260620 15:16:17.744606 1582 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:17.744732 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:17.745110 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:17.749155 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "9029ef89b2c7430281faad74dc369ea5"
format_stamp: "Formatted at 2026-06-20 15:16:17 on dist-test-slave-7s64"
server_key: "4df6289ebd1d25194410a20f90605651"
server_key_iv: "80dc6daca88357ddfa84cfeb647da504"
server_key_version: "encryptionkey@0"
I20260620 15:16:17.750190 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "9029ef89b2c7430281faad74dc369ea5"
format_stamp: "Formatted at 2026-06-20 15:16:17 on dist-test-slave-7s64"
server_key: "4df6289ebd1d25194410a20f90605651"
server_key_iv: "80dc6daca88357ddfa84cfeb647da504"
server_key_version: "encryptionkey@0"
I20260620 15:16:17.757249 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 15:16:17.761857 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:17.763463 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:16:17.763648 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "9029ef89b2c7430281faad74dc369ea5"
format_stamp: "Formatted at 2026-06-20 15:16:17 on dist-test-slave-7s64"
server_key: "4df6289ebd1d25194410a20f90605651"
server_key_iv: "80dc6daca88357ddfa84cfeb647da504"
server_key_version: "encryptionkey@0"
I20260620 15:16:17.763890 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:17.788175 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:17.792668 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:17.792945 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:17.795542 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:17.795653 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:17.795818 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:17.795893 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:16:17 dist-test-slave-7s64 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:17.825480 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:17.797092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37043 (local address 127.0.97.65:39019)
0620 15:16:17.798144 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:17.798195 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:17.800302 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:17.801390 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:17.801407 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:17.802408 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:17.803462 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:17.803494 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:17.805255 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:17.805262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:17.808126 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:17.808144 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:17.809257 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:17.809270 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:17.809646 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:17.810753 (+ 1107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:17.810800 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:17.814889 (+ 4089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:17.819275 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:17.819295 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:17.819306 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:17.820345 (+ 1039us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:17.820890 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:17.820897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:17.820902 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:17.821195 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:17.822014 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:17.822024 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:17.822592 (+ 568us) client_negotiation.cc:769] Sending connection context
0620 15:16:17.823000 (+ 408us) client_negotiation.cc:240] Negotiation successful
0620 15:16:17.823700 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":508,"threads_started":1}
I20260620 15:16:17.826664 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:17.796272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:39019 (local address 127.0.97.126:37043)
0620 15:16:17.797893 (+ 1621us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:17.797897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:17.798313 (+ 416us) server_negotiation.cc:407] Connection header received
0620 15:16:17.800802 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:17.800809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:17.800888 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:17.801000 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:17.803753 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:17.805044 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:17.808365 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:17.808948 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:17.809088 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:17.815193 (+ 6105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:17.815200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:17.815206 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:17.815253 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:17.819092 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:17.820558 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:17.820566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:17.820570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:17.820726 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:17.821356 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:17.821361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:17.821366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:17.821763 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:17.821917 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:17.825885 (+ 3968us) server_negotiation.cc:299] Negotiation successful
0620 15:16:17.826174 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1509,"thread_start_us":1426,"threads_started":1}
I20260620 15:16:17.834945 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:40867
I20260620 15:16:17.835031 1716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:40867 every 8 connection(s)
I20260620 15:16:17.836319 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:17.838721 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37043
I20260620 15:16:17.838950 1718 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:17.839709 1718 heartbeater.cc:511] Master 127.0.97.126:37043 requested a full tablet report, sending...
I20260620 15:16:17.841310 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 1582
I20260620 15:16:17.841496 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:17.842316 389 external_mini_cluster.cc:1428] Setting key 67dc02b497370f336e3a8825ba4a7c7b
I20260620 15:16:17.842298 1511 ts_manager.cc:194] Registered new tserver with Master: 9029ef89b2c7430281faad74dc369ea5 (127.0.97.65:40867)
I20260620 15:16:17.844399 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:39019
I20260620 15:16:17.849485 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:17.835760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:44573 (local address 127.0.97.126:37043)
0620 15:16:17.835954 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:17.835962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:17.836627 (+ 665us) server_negotiation.cc:407] Connection header received
0620 15:16:17.836835 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:17.836843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:17.836935 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:17.837038 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:17.838334 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:17.839310 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:17.841216 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:17.841893 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:17.842034 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:17.844101 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:17.844108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:17.844112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:17.844147 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:17.847022 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:17.847657 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:17.847663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:17.847667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:17.847814 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:17.848156 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:17.848162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:17.848166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:17.848401 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:17.848565 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:17.848952 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 15:16:17.849153 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:16:17.849659 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:17.835772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37043 (local address 127.0.97.65:44573)
0620 15:16:17.836516 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:17.836534 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:17.836696 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:17.837105 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:17.837114 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:17.837906 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:17.838200 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:17.838209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:17.839449 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:17.839454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:17.841054 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:17.841065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:17.842089 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:17.842096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:17.842267 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:17.842934 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:17.842958 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:17.843909 (+ 951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:17.847149 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:17.847156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:17.847160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:17.847542 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:17.847919 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:17.847925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:17.847928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:17.848054 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:17.848544 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:17.848549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:17.848715 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 15:16:17.848860 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 15:16:17.849150 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":379,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:16:17.881739 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37043
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:18.191287 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:18.191529 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:18.191632 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:18.198532 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:18.198622 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:18.198700 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:16:18.198769 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:18.207805 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:37043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:18.208902 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:18.210065 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:18.222302 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:18.222312 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:18.222891 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:18.223305 1722 server_base.cc:1034] running on GCE node
I20260620 15:16:18.223943 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:18.224902 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:18.226130 1722 hybrid_clock.cc:648] HybridClock initialized: now 1781968578226054 us; error 53 us; skew 500 ppm
I20260620 15:16:18.226418 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:18 dist-test-slave-7s64 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:18.229923 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:16:18.235437 1722 webserver.cc:460] Webserver started at http://127.0.97.66:33915/ using document root <none> and password file <none>
I20260620 15:16:18.236109 1722 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:18.236218 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:18.236594 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:18.239346 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "0431211c57bb490c8ac41ef5c106e546"
format_stamp: "Formatted at 2026-06-20 15:16:18 on dist-test-slave-7s64"
server_key: "8b3b5fd023f7ea88ff70fa91610d2d0a"
server_key_iv: "91abe5986e79118ef8c4e1ac856fa9f7"
server_key_version: "encryptionkey@0"
I20260620 15:16:18.240116 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "0431211c57bb490c8ac41ef5c106e546"
format_stamp: "Formatted at 2026-06-20 15:16:18 on dist-test-slave-7s64"
server_key: "8b3b5fd023f7ea88ff70fa91610d2d0a"
server_key_iv: "91abe5986e79118ef8c4e1ac856fa9f7"
server_key_version: "encryptionkey@0"
I20260620 15:16:18.245000 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 15:16:18.248332 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:18.249742 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:18.249917 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "0431211c57bb490c8ac41ef5c106e546"
format_stamp: "Formatted at 2026-06-20 15:16:18 on dist-test-slave-7s64"
server_key: "8b3b5fd023f7ea88ff70fa91610d2d0a"
server_key_iv: "91abe5986e79118ef8c4e1ac856fa9f7"
server_key_version: "encryptionkey@0"
I20260620 15:16:18.250129 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:18.268872 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:18.273043 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:18.273260 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:18.275908 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:18.275985 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:18.276122 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:18.276204 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:16:18 dist-test-slave-7s64 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:18.301455 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.276852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52935 (local address 127.0.97.126:37043)
0620 15:16:18.277065 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:18.277073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:18.278703 (+ 1630us) server_negotiation.cc:407] Connection header received
0620 15:16:18.281432 (+ 2729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:18.281439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:18.281577 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:18.281697 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:18.284095 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.285489 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.288398 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.289236 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.289394 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.293528 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:18.293537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:18.293544 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:18.293595 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:18.297430 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:18.298380 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:18.298387 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:18.298391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:18.298539 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:18.299174 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:18.299181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:18.299187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:18.299647 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:18.299806 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:18.301014 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0620 15:16:18.301208 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:16:18.303421 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.277466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37043 (local address 127.0.97.66:52935)
0620 15:16:18.278518 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:18.278569 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:18.281105 (+ 2536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:18.281989 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:18.282026 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:18.282901 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:18.283868 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.283896 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.285690 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.285697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.288222 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.288245 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.289587 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.289593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.289763 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.290439 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:18.290460 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:18.293298 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:18.297612 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:18.297621 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:18.297644 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:18.298246 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:18.298676 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:18.298681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:18.298684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:18.299029 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:18.299903 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:18.299918 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:18.300475 (+ 557us) client_negotiation.cc:769] Sending connection context
0620 15:16:18.300905 (+ 430us) client_negotiation.cc:240] Negotiation successful
0620 15:16:18.301601 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":495,"threads_started":1}
I20260620 15:16:18.320999 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39201
I20260620 15:16:18.321020 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39201 every 8 connection(s)
I20260620 15:16:18.322649 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:18.324929 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37043
I20260620 15:16:18.325161 1857 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:18.325893 1857 heartbeater.cc:511] Master 127.0.97.126:37043 requested a full tablet report, sending...
I20260620 15:16:18.326573 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.311040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:43113 (local address 127.0.97.126:37043)
0620 15:16:18.311257 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:18.311266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:18.311827 (+ 561us) server_negotiation.cc:407] Connection header received
0620 15:16:18.312034 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:18.312039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:18.312119 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:18.312245 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:18.313367 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.314389 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.316668 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.317490 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.317648 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.319806 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:18.319815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:18.319823 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:18.319888 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:18.323481 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:18.324430 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:18.324433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:18.324435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:18.324637 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:18.325021 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:18.325028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:18.325032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:18.325412 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:18.325562 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:18.326007 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 15:16:18.326229 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:16:18.326727 1832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.311008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37043 (local address 127.0.97.66:43113)
0620 15:16:18.311707 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:18.311732 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:18.311905 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:18.312254 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:18.312257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:18.312733 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:18.313166 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.313186 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.314578 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.314582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.316419 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.316431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.317714 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.317723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.317938 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.318665 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:18.318694 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:18.319585 (+ 891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:18.323660 (+ 4075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:18.323665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:18.323668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:18.324269 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:18.324751 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:18.324756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:18.324760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:18.324905 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:18.325528 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:18.325532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:18.325772 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 15:16:18.325923 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 15:16:18.326235 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":19,"thread_start_us":382,"threads_started":1}
I20260620 15:16:18.327468 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 1722
I20260620 15:16:18.327562 1510 ts_manager.cc:194] Registered new tserver with Master: 0431211c57bb490c8ac41ef5c106e546 (127.0.97.66:39201)
I20260620 15:16:18.327632 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:18.328049 389 external_mini_cluster.cc:1428] Setting key a11175fa09ddc0a2d55ad0bb4b270720
I20260620 15:16:18.328604 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:52935
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:16:18.367501 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37043
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:18.682596 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:18.682888 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:18.683063 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:18.690140 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:18.690240 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:18.690320 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:16:18.690397 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:18.699800 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:37043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:18.701010 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:18.702354 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:18.715059 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:18.715088 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:18.715149 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:18.715657 1861 server_base.cc:1034] running on GCE node
I20260620 15:16:18.716284 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:18.717233 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:18.718487 1861 hybrid_clock.cc:648] HybridClock initialized: now 1781968578718408 us; error 51 us; skew 500 ppm
I20260620 15:16:18.718801 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:18 dist-test-slave-7s64 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:18.721773 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:16:18.723091 1861 webserver.cc:460] Webserver started at http://127.0.97.67:40867/ using document root <none> and password file <none>
I20260620 15:16:18.723735 1861 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:18.723860 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:18.724161 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:18.726900 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "0ba70768088548b0babb46364900364f"
format_stamp: "Formatted at 2026-06-20 15:16:18 on dist-test-slave-7s64"
server_key: "9fc2c28d2110fcef0e0cdb361e2dc24e"
server_key_iv: "f04dfe065f205ec9eff847bd3fe25fb1"
server_key_version: "encryptionkey@0"
I20260620 15:16:18.727632 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "0ba70768088548b0babb46364900364f"
format_stamp: "Formatted at 2026-06-20 15:16:18 on dist-test-slave-7s64"
server_key: "9fc2c28d2110fcef0e0cdb361e2dc24e"
server_key_iv: "f04dfe065f205ec9eff847bd3fe25fb1"
server_key_version: "encryptionkey@0"
I20260620 15:16:18.732440 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 15:16:18.735723 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:18.737035 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:18.737205 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "0ba70768088548b0babb46364900364f"
format_stamp: "Formatted at 2026-06-20 15:16:18 on dist-test-slave-7s64"
server_key: "9fc2c28d2110fcef0e0cdb361e2dc24e"
server_key_iv: "f04dfe065f205ec9eff847bd3fe25fb1"
server_key_version: "encryptionkey@0"
I20260620 15:16:18.737414 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:18.748759 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:18.752384 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:18.752655 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:18.755218 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:18.755330 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:18.755452 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:18.755524 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:16:18 dist-test-slave-7s64 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:18.779850 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.756549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37043 (local address 127.0.97.67:59273)
0620 15:16:18.757388 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:18.757473 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:18.759350 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:18.760050 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:18.760058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:18.760689 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:18.761617 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.761635 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.763010 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.763022 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.765915 (+ 2893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.765932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.766906 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.766914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.767099 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.767720 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:18.767748 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:18.771053 (+ 3305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:18.774362 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:18.774374 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:18.774395 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:18.775127 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:18.775540 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:18.775548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:18.775554 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:18.775843 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:18.776336 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:18.776347 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:18.776857 (+ 510us) client_negotiation.cc:769] Sending connection context
0620 15:16:18.777199 (+ 342us) client_negotiation.cc:240] Negotiation successful
0620 15:16:18.777867 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":392,"threads_started":1}
I20260620 15:16:18.780433 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.755817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59273 (local address 127.0.97.126:37043)
0620 15:16:18.755975 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:18.755992 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:18.757586 (+ 1594us) server_negotiation.cc:407] Connection header received
0620 15:16:18.759622 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:18.759628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:18.759710 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:18.759834 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:18.761794 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.762777 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.766075 (+ 3298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.766639 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.766780 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.771326 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:18.771335 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:18.771343 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:18.771384 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:18.774230 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:18.775285 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:18.775289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:18.775291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:18.775419 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:18.775962 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:18.775965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:18.775967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:18.776210 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:18.776315 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:18.779632 (+ 3317us) server_negotiation.cc:299] Negotiation successful
0620 15:16:18.779936 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 15:16:18.801442 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.786209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37043 (local address 127.0.97.67:39379)
0620 15:16:18.786942 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:18.786966 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:18.787125 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:18.787434 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:18.787442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:18.788167 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:18.788637 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.788653 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.789573 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.789581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.791655 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.791673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.792384 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.792394 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.792747 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.793551 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:18.793570 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:18.794796 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:18.798550 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:18.798553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:18.798556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:18.799137 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:18.799588 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:18.799599 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:18.799606 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:18.799807 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:18.800576 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:18.800583 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:18.800787 (+ 204us) client_negotiation.cc:769] Sending connection context
0620 15:16:18.800905 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 15:16:18.801123 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":357,"threads_started":1}
I20260620 15:16:18.801635 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.786193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:39379 (local address 127.0.97.126:37043)
0620 15:16:18.786381 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:18.786388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:18.787060 (+ 672us) server_negotiation.cc:407] Connection header received
0620 15:16:18.787247 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:18.787251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:18.787303 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:18.787372 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:18.788787 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.789446 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.791812 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.792190 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.792281 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.794993 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:18.794998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:18.795001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:18.795027 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:18.798394 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:18.799267 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:18.799275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:18.799280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:18.799453 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:18.799999 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:18.800007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:18.800014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:18.800405 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:18.800604 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:18.801042 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 15:16:18.801282 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:16:18.812621 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:37577
I20260620 15:16:18.812649 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:37577 every 8 connection(s)
I20260620 15:16:18.813814 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:18.815728 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37043
I20260620 15:16:18.815990 1996 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:18.816787 1996 heartbeater.cc:511] Master 127.0.97.126:37043 requested a full tablet report, sending...
I20260620 15:16:18.818199 1511 ts_manager.cc:194] Registered new tserver with Master: 0ba70768088548b0babb46364900364f (127.0.97.67:37577)
I20260620 15:16:18.818982 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:59273
I20260620 15:16:18.823387 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 1861
I20260620 15:16:18.823513 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:18.823875 389 external_mini_cluster.cc:1428] Setting key b5e8e8a70b3ad6c52426f11c3407e864
I20260620 15:16:18.826632 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:16:18.838449 1713 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.97.65:40867
I20260620 15:16:18.848552 1718 heartbeater.cc:503] Master 127.0.97.126:37043 was elected leader, sending a full tablet report...
Jun 20 15:16:18 dist-test-slave-7s64 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:16:18 dist-test-slave-7s64 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:18.872584 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.858190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.0.97.126:37043)
0620 15:16:18.858442 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:18.858448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:18.858611 (+ 163us) server_negotiation.cc:407] Connection header received
0620 15:16:18.858948 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:18.858957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:18.859039 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:18.859197 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:18.860787 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.861578 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.862930 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.863372 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.863525 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.867616 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:18.867621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:18.867624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:18.867656 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:18.870292 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:18.870856 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:18.870860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:18.870862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:18.870967 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:18.871345 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:18.871348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:18.871351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:18.871668 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:18.871886 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:18.872103 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 15:16:18.872274 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 15:16:18.876459 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:16:18.883337 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:18.907756 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.895019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39201 (local address 127.0.0.1:42822)
0620 15:16:18.896315 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:18.896336 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:18.896459 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:18.897781 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:18.897792 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:18.897841 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:18.898410 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.898425 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.901060 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.901069 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.904155 (+ 3086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.904174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.906716 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.906725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.907069 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.907143 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 15:16:18.907253 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 15:16:18.907396 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"thread_start_us":12021,"threads_started":1}
I20260620 15:16:18.908941 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.896021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.0.97.66:39201)
0620 15:16:18.897023 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:18.897035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:18.897070 (+ 35us) server_negotiation.cc:407] Connection header received
0620 15:16:18.897186 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:18.897197 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:18.897512 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:18.898685 (+ 1173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:18.898921 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.900877 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.904419 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.906444 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.908182 (+ 1738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.908389 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:18.908450 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 15:16:18.908620 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":718,"threads_started":1}
I20260620 15:16:18.912626 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 412e427d6496475e9b0842b71df83eb5 (DEFAULT_TABLE table=test-table [id=117a4a3234414063b94df6a5a4b6379a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:18.914120 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412e427d6496475e9b0842b71df83eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:18.914924 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.895676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:40867 (local address 127.0.0.1:35234)
0620 15:16:18.907393 (+ 11717us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:18.907416 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:18.907572 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:18.908206 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:18.908213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:18.908241 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:18.908598 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.908630 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.910320 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.910325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.912407 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.912421 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.914164 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.914170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.914334 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.914380 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 15:16:18.914457 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 15:16:18.914579 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11554,"thread_start_us":1279,"threads_started":1}
I20260620 15:16:18.915282 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.894528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37577 (local address 127.0.0.1:41692)
0620 15:16:18.895348 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:18.895405 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:18.895601 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:18.908810 (+ 13209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:18.908814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:18.908837 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:18.909194 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.909207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.911433 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.911438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.913148 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:18.913157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.914519 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.914525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:18.914714 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.914744 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:16:18.914846 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:16:18.914970 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":412,"threads_started":1}
I20260620 15:16:18.915282 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.896572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.0.97.65:40867)
0620 15:16:18.906687 (+ 10115us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:18.906695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:18.907544 (+ 849us) server_negotiation.cc:407] Connection header received
0620 15:16:18.907750 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:18.907758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:18.907994 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:18.908827 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:18.909031 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.910176 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.912599 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.913965 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.914098 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.914296 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:18.914804 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 15:16:18.914972 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9943,"thread_start_us":10104,"threads_started":1}
I20260620 15:16:18.916117 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:18.894825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.0.97.67:37577)
0620 15:16:18.908206 (+ 13381us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:18.908214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:18.908244 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:16:18.908342 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:18.908349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:18.908599 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:18.909419 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:18.909712 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.911280 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.913291 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:18.914318 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:18.914458 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:18.914662 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:18.915646 (+ 984us) server_negotiation.cc:299] Negotiation successful
0620 15:16:18.915811 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13229,"thread_start_us":13044,"threads_started":1}
I20260620 15:16:18.918744 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 412e427d6496475e9b0842b71df83eb5 (DEFAULT_TABLE table=test-table [id=117a4a3234414063b94df6a5a4b6379a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:18.919332 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 412e427d6496475e9b0842b71df83eb5 (DEFAULT_TABLE table=test-table [id=117a4a3234414063b94df6a5a4b6379a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:18.920101 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412e427d6496475e9b0842b71df83eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:18.920655 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412e427d6496475e9b0842b71df83eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:18.926507 2015 tablet_bootstrap.cc:492] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: Bootstrap starting.
I20260620 15:16:18.929745 2015 tablet_bootstrap.cc:654] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:18.930774 2015 log.cc:824] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:18.933777 2015 tablet_bootstrap.cc:492] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: No bootstrap required, opened a new log
I20260620 15:16:18.934125 2015 ts_tablet_manager.cc:1397] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:16:18.934252 2017 tablet_bootstrap.cc:492] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: Bootstrap starting.
I20260620 15:16:18.937184 2018 tablet_bootstrap.cc:492] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: Bootstrap starting.
I20260620 15:16:18.937654 2017 tablet_bootstrap.cc:654] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:18.938951 2017 log.cc:824] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:18.940758 2018 tablet_bootstrap.cc:654] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:18.941920 2018 log.cc:824] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:18.941969 2017 tablet_bootstrap.cc:492] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: No bootstrap required, opened a new log
I20260620 15:16:18.942203 2017 ts_tablet_manager.cc:1397] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260620 15:16:18.943776 2015 raft_consensus.cc:348] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } }
I20260620 15:16:18.944130 2015 raft_consensus.cc:374] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:18.944231 2015 raft_consensus.cc:729] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0431211c57bb490c8ac41ef5c106e546, State: Initialized, Role: FOLLOWER
I20260620 15:16:18.944849 2015 consensus_queue.cc:260] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } }
I20260620 15:16:18.945338 2018 tablet_bootstrap.cc:492] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: No bootstrap required, opened a new log
I20260620 15:16:18.945580 2018 ts_tablet_manager.cc:1397] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 15:16:18.945909 1857 heartbeater.cc:503] Master 127.0.97.126:37043 was elected leader, sending a full tablet report...
I20260620 15:16:18.946460 2015 ts_tablet_manager.cc:1428] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260620 15:16:18.953403 2017 raft_consensus.cc:348] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } }
I20260620 15:16:18.953817 2017 raft_consensus.cc:374] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:18.953917 2017 raft_consensus.cc:729] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9029ef89b2c7430281faad74dc369ea5, State: Initialized, Role: FOLLOWER
I20260620 15:16:18.954487 2017 consensus_queue.cc:260] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } }
I20260620 15:16:18.956128 2017 ts_tablet_manager.cc:1428] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260620 15:16:18.956357 2018 raft_consensus.cc:348] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } }
I20260620 15:16:18.956799 2018 raft_consensus.cc:374] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:18.956895 2018 raft_consensus.cc:729] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba70768088548b0babb46364900364f, State: Initialized, Role: FOLLOWER
I20260620 15:16:18.957361 2018 consensus_queue.cc:260] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } }
I20260620 15:16:18.958405 1996 heartbeater.cc:503] Master 127.0.97.126:37043 was elected leader, sending a full tablet report...
I20260620 15:16:18.958928 2018 ts_tablet_manager.cc:1428] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260620 15:16:19.065472 1997 tablet.cc:2369] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:16:19.074855 1858 tablet.cc:2369] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:16:19.089138 1719 tablet.cc:2369] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:16:19.255468 2022 raft_consensus.cc:482] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:19.255928 2022 raft_consensus.cc:504] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } }
I20260620 15:16:19.257845 2022 leader_election.cc:288] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ba70768088548b0babb46364900364f (127.0.97.67:37577), 0431211c57bb490c8ac41ef5c106e546 (127.0.97.66:39201)
I20260620 15:16:19.264618 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.258401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:59591 (local address 127.0.97.67:37577)
0620 15:16:19.258618 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.258627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.259291 (+ 664us) server_negotiation.cc:407] Connection header received
0620 15:16:19.259476 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.259483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.259543 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.259637 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:19.260259 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.261006 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.262883 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.263621 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.263769 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.263891 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.264198 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.264318 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 15:16:19.264632 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.258328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57787 (local address 127.0.97.66:39201)
0620 15:16:19.258577 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.258586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.259206 (+ 620us) server_negotiation.cc:407] Connection header received
0620 15:16:19.259432 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.259436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.259506 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.259605 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:19.260253 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.261016 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.262895 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.263620 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.263770 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.263891 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.264235 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.264354 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:16:19.265267 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.258110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39201 (local address 127.0.97.65:57787)
0620 15:16:19.259100 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:19.259115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:19.259274 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:19.259619 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:19.259624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:19.259647 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:19.260085 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.260094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.261138 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.261146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.262743 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.262754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.263911 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.263919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.264069 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.264090 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:16:19.264193 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:16:19.264312 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":6,"thread_start_us":708,"threads_started":1}
I20260620 15:16:19.265286 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.258264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37577 (local address 127.0.97.65:59591)
0620 15:16:19.259188 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:19.259202 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:19.259345 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:19.259651 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:19.259655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:19.259678 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:19.260100 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.260109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.261137 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.261142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.262735 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.262746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.263855 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.263862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.264049 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.264071 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:16:19.264174 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:16:19.264297 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":93,"thread_start_us":634,"threads_started":1}
I20260620 15:16:19.266575 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412e427d6496475e9b0842b71df83eb5" candidate_uuid: "9029ef89b2c7430281faad74dc369ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba70768088548b0babb46364900364f" is_pre_election: true
I20260620 15:16:19.266575 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412e427d6496475e9b0842b71df83eb5" candidate_uuid: "9029ef89b2c7430281faad74dc369ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431211c57bb490c8ac41ef5c106e546" is_pre_election: true
I20260620 15:16:19.267180 1951 raft_consensus.cc:2454] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9029ef89b2c7430281faad74dc369ea5 in term 0.
I20260620 15:16:19.267190 1807 raft_consensus.cc:2454] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9029ef89b2c7430281faad74dc369ea5 in term 0.
I20260620 15:16:19.267897 1601 leader_election.cc:302] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba70768088548b0babb46364900364f, 9029ef89b2c7430281faad74dc369ea5; no voters:
I20260620 15:16:19.268399 2022 raft_consensus.cc:2781] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:19.268575 2022 raft_consensus.cc:482] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:19.268684 2022 raft_consensus.cc:3037] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:19.270970 2022 raft_consensus.cc:504] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } }
I20260620 15:16:19.271538 2022 leader_election.cc:288] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [CANDIDATE]: Term 1 election: Requested vote from peers 0ba70768088548b0babb46364900364f (127.0.97.67:37577), 0431211c57bb490c8ac41ef5c106e546 (127.0.97.66:39201)
I20260620 15:16:19.272038 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412e427d6496475e9b0842b71df83eb5" candidate_uuid: "9029ef89b2c7430281faad74dc369ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba70768088548b0babb46364900364f"
I20260620 15:16:19.272038 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412e427d6496475e9b0842b71df83eb5" candidate_uuid: "9029ef89b2c7430281faad74dc369ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431211c57bb490c8ac41ef5c106e546"
I20260620 15:16:19.272241 1807 raft_consensus.cc:3037] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:19.272351 1951 raft_consensus.cc:3037] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:19.275445 1951 raft_consensus.cc:2454] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9029ef89b2c7430281faad74dc369ea5 in term 1.
I20260620 15:16:19.275446 1807 raft_consensus.cc:2454] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9029ef89b2c7430281faad74dc369ea5 in term 1.
I20260620 15:16:19.275833 1599 leader_election.cc:302] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0431211c57bb490c8ac41ef5c106e546, 9029ef89b2c7430281faad74dc369ea5; no voters:
I20260620 15:16:19.276144 2022 raft_consensus.cc:2781] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:19.276573 2022 raft_consensus.cc:686] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 1 LEADER]: Becoming Leader. State: Replica: 9029ef89b2c7430281faad74dc369ea5, State: Running, Role: LEADER
I20260620 15:16:19.277032 2022 consensus_queue.cc:237] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } }
I20260620 15:16:19.281358 1511 catalog_manager.cc:5495] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9029ef89b2c7430281faad74dc369ea5 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "9029ef89b2c7430281faad74dc369ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } health_report { overall_health: UNKNOWN } } }
I20260620 15:16:19.364287 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.358872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.0.97.65:40867)
0620 15:16:19.359076 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.359084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.359207 (+ 123us) server_negotiation.cc:407] Connection header received
0620 15:16:19.359539 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.359543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.359634 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.359792 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:19.360159 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.360937 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.362266 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.362645 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.362776 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.363251 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:19.363709 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:19.363858 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.363954 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.364027 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:16:19.371855 1951 raft_consensus.cc:1261] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f [term 1 FOLLOWER]: Refusing update from remote peer 9029ef89b2c7430281faad74dc369ea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:19.372062 1807 raft_consensus.cc:1261] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546 [term 1 FOLLOWER]: Refusing update from remote peer 9029ef89b2c7430281faad74dc369ea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:19.372977 2022 consensus_queue.cc:1035] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:19.373247 2026 consensus_queue.cc:1035] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:19.384349 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298943301097558016 to 7298943300722335744. Current Snapshot: MvccSnapshot[applied={T|T < 7298943301097558016}]
I20260620 15:16:19.404903 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:19.405114 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:19.408610 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5: Sending DeleteTablet for 3 replicas of tablet 412e427d6496475e9b0842b71df83eb5
I20260620 15:16:19.409693 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 412e427d6496475e9b0842b71df83eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:19 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:41692
I20260620 15:16:19.410001 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 412e427d6496475e9b0842b71df83eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:19 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:42822
I20260620 15:16:19.411087 2040 tablet_replica.cc:330] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: stopping tablet replica
I20260620 15:16:19.411273 2041 tablet_replica.cc:330] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: stopping tablet replica
I20260620 15:16:19.411502 2040 raft_consensus.cc:2229] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:19.411581 2041 raft_consensus.cc:2229] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:19.411942 2040 raft_consensus.cc:2258] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:19.411964 2041 raft_consensus.cc:2258] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:19.412937 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 412e427d6496475e9b0842b71df83eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:19 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:35234
I20260620 15:16:19.413985 2040 ts_tablet_manager.cc:1905] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:19.414186 2041 ts_tablet_manager.cc:1905] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:19.416286 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:16:19.417595 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:19.421373 2041 ts_tablet_manager.cc:1918] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:16:19.421522 2041 log.cc:1196] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/412e427d6496475e9b0842b71df83eb5
I20260620 15:16:19.422030 2041 ts_tablet_manager.cc:1939] T 412e427d6496475e9b0842b71df83eb5 P 0431211c57bb490c8ac41ef5c106e546: Deleting consensus metadata
I20260620 15:16:19.422974 2040 ts_tablet_manager.cc:1918] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:16:19.423108 2040 log.cc:1196] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/412e427d6496475e9b0842b71df83eb5
I20260620 15:16:19.423583 2040 ts_tablet_manager.cc:1939] T 412e427d6496475e9b0842b71df83eb5 P 0ba70768088548b0babb46364900364f: Deleting consensus metadata
I20260620 15:16:19.423846 1496 catalog_manager.cc:4841] TS 0431211c57bb490c8ac41ef5c106e546 (127.0.97.66:39201): tablet 412e427d6496475e9b0842b71df83eb5 (table test-table [id=117a4a3234414063b94df6a5a4b6379a]) successfully deleted
I20260620 15:16:19.425174 1498 catalog_manager.cc:4841] TS 0ba70768088548b0babb46364900364f (127.0.97.67:37577): tablet 412e427d6496475e9b0842b71df83eb5 (table test-table [id=117a4a3234414063b94df6a5a4b6379a]) successfully deleted
I20260620 15:16:19.427251 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 59425e8d0f454eb3a6269dff69780154 (DEFAULT_TABLE table=test-table [id=175b1ac3371d489c98007db0194a58b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:19.427347 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 59425e8d0f454eb3a6269dff69780154 (DEFAULT_TABLE table=test-table [id=175b1ac3371d489c98007db0194a58b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:19.427649 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59425e8d0f454eb3a6269dff69780154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:19.427874 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59425e8d0f454eb3a6269dff69780154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:19.430725 2042 tablet_replica.cc:330] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: stopping tablet replica
I20260620 15:16:19.431140 2042 raft_consensus.cc:2229] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:16:19.431838 2042 raft_consensus.cc:2258] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:19.432984 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 59425e8d0f454eb3a6269dff69780154 (DEFAULT_TABLE table=test-table [id=175b1ac3371d489c98007db0194a58b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:19.433378 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59425e8d0f454eb3a6269dff69780154. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:19.434865 2042 ts_tablet_manager.cc:1905] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:19.435940 2017 tablet_bootstrap.cc:492] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5: Bootstrap starting.
I20260620 15:16:19.438300 2017 tablet_bootstrap.cc:654] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:19.438409 2018 tablet_bootstrap.cc:492] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f: Bootstrap starting.
I20260620 15:16:19.440043 2018 tablet_bootstrap.cc:654] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:19.440083 2015 tablet_bootstrap.cc:492] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546: Bootstrap starting.
I20260620 15:16:19.441668 2015 tablet_bootstrap.cc:654] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:19.450840 2015 tablet_bootstrap.cc:492] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546: No bootstrap required, opened a new log
I20260620 15:16:19.450978 2015 ts_tablet_manager.cc:1397] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 15:16:19.451895 2015 raft_consensus.cc:348] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.452189 2015 raft_consensus.cc:374] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:19.452302 2015 raft_consensus.cc:729] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0431211c57bb490c8ac41ef5c106e546, State: Initialized, Role: FOLLOWER
I20260620 15:16:19.452692 2015 consensus_queue.cc:260] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.455010 2042 ts_tablet_manager.cc:1918] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:16:19.455263 2042 log.cc:1196] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/412e427d6496475e9b0842b71df83eb5
I20260620 15:16:19.456071 2042 ts_tablet_manager.cc:1939] T 412e427d6496475e9b0842b71df83eb5 P 9029ef89b2c7430281faad74dc369ea5: Deleting consensus metadata
I20260620 15:16:19.456813 2015 ts_tablet_manager.cc:1428] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260620 15:16:19.458864 1498 catalog_manager.cc:4841] TS 9029ef89b2c7430281faad74dc369ea5 (127.0.97.65:40867): tablet 412e427d6496475e9b0842b71df83eb5 (table test-table [id=117a4a3234414063b94df6a5a4b6379a]) successfully deleted
I20260620 15:16:19.460925 2018 tablet_bootstrap.cc:492] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f: No bootstrap required, opened a new log
I20260620 15:16:19.461086 2018 ts_tablet_manager.cc:1397] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.015s
I20260620 15:16:19.461488 2017 tablet_bootstrap.cc:492] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5: No bootstrap required, opened a new log
I20260620 15:16:19.461623 2017 ts_tablet_manager.cc:1397] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.020s
I20260620 15:16:19.461884 2018 raft_consensus.cc:348] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.462093 2018 raft_consensus.cc:374] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:19.462189 2018 raft_consensus.cc:729] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba70768088548b0babb46364900364f, State: Initialized, Role: FOLLOWER
I20260620 15:16:19.462409 2017 raft_consensus.cc:348] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.462476 2018 consensus_queue.cc:260] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.462625 2017 raft_consensus.cc:374] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:19.462713 2017 raft_consensus.cc:729] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9029ef89b2c7430281faad74dc369ea5, State: Initialized, Role: FOLLOWER
I20260620 15:16:19.463017 2017 consensus_queue.cc:260] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.463446 2018 ts_tablet_manager.cc:1428] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:16:19.464053 2017 ts_tablet_manager.cc:1428] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260620 15:16:19.531869 2023 raft_consensus.cc:482] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:19.532487 2023 raft_consensus.cc:504] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.534029 2023 leader_election.cc:288] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9029ef89b2c7430281faad74dc369ea5 (127.0.97.65:40867), 0431211c57bb490c8ac41ef5c106e546 (127.0.97.66:39201)
I20260620 15:16:19.541873 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.534627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54067 (local address 127.0.97.65:40867)
0620 15:16:19.534807 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.534813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.535248 (+ 435us) server_negotiation.cc:407] Connection header received
0620 15:16:19.535519 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.535526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.535601 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.535663 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:19.536434 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.537511 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.539531 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.540873 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.541016 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.541110 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.541458 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.541595 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:16:19.542241 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.534528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:60179 (local address 127.0.97.66:39201)
0620 15:16:19.534725 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.534733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.536371 (+ 1638us) server_negotiation.cc:407] Connection header received
0620 15:16:19.536540 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.536546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.536633 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.536727 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:19.537274 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.538351 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.540229 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.541304 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.541435 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.541516 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.541863 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.541991 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:16:19.542904 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.534293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39201 (local address 127.0.97.67:60179)
0620 15:16:19.536251 (+ 1958us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:19.536272 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:19.536395 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:19.536767 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:19.536772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:19.536798 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:19.537109 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.537120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.538480 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.538486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.540081 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.540091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.541499 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.541505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.541679 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.541713 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:16:19.541817 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:16:19.541946 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1835,"thread_start_us":1412,"threads_started":1}
I20260620 15:16:19.543078 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.534258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:40867 (local address 127.0.97.67:54067)
0620 15:16:19.535142 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:19.535160 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:19.535357 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:19.535865 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:19.535871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:19.535906 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:19.536265 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.536278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.537639 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.537644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.539366 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.539380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.541060 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.541066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.541249 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.541285 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:16:19.542136 (+ 851us) client_negotiation.cc:240] Negotiation successful
0620 15:16:19.542283 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":1,"thread_start_us":1716,"threads_started":1}
I20260620 15:16:19.543560 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59425e8d0f454eb3a6269dff69780154" candidate_uuid: "0ba70768088548b0babb46364900364f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431211c57bb490c8ac41ef5c106e546" is_pre_election: true
I20260620 15:16:19.543805 1807 raft_consensus.cc:2454] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ba70768088548b0babb46364900364f in term 0.
I20260620 15:16:19.543896 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59425e8d0f454eb3a6269dff69780154" candidate_uuid: "0ba70768088548b0babb46364900364f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9029ef89b2c7430281faad74dc369ea5" is_pre_election: true
I20260620 15:16:19.544262 1878 leader_election.cc:302] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0431211c57bb490c8ac41ef5c106e546, 0ba70768088548b0babb46364900364f; no voters:
I20260620 15:16:19.544349 1668 raft_consensus.cc:2454] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ba70768088548b0babb46364900364f in term 0.
I20260620 15:16:19.544723 2023 raft_consensus.cc:2781] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:19.544871 2023 raft_consensus.cc:482] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:19.544958 2023 raft_consensus.cc:3037] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:19.547087 2023 raft_consensus.cc:504] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.547654 2023 leader_election.cc:288] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [CANDIDATE]: Term 1 election: Requested vote from peers 9029ef89b2c7430281faad74dc369ea5 (127.0.97.65:40867), 0431211c57bb490c8ac41ef5c106e546 (127.0.97.66:39201)
I20260620 15:16:19.547977 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59425e8d0f454eb3a6269dff69780154" candidate_uuid: "0ba70768088548b0babb46364900364f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9029ef89b2c7430281faad74dc369ea5"
I20260620 15:16:19.548175 1668 raft_consensus.cc:3037] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:19.548221 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59425e8d0f454eb3a6269dff69780154" candidate_uuid: "0ba70768088548b0babb46364900364f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431211c57bb490c8ac41ef5c106e546"
I20260620 15:16:19.548458 1807 raft_consensus.cc:3037] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:19.550379 1807 raft_consensus.cc:2454] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ba70768088548b0babb46364900364f in term 1.
I20260620 15:16:19.550712 1878 leader_election.cc:302] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0431211c57bb490c8ac41ef5c106e546, 0ba70768088548b0babb46364900364f; no voters:
I20260620 15:16:19.550976 2023 raft_consensus.cc:2781] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:19.551492 2023 raft_consensus.cc:686] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 1 LEADER]: Becoming Leader. State: Replica: 0ba70768088548b0babb46364900364f, State: Running, Role: LEADER
I20260620 15:16:19.551856 2023 consensus_queue.cc:237] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.551968 1668 raft_consensus.cc:2454] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ba70768088548b0babb46364900364f in term 1.
I20260620 15:16:19.555260 1511 catalog_manager.cc:5495] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ba70768088548b0babb46364900364f (127.0.97.67). New cstate: current_term: 1 leader_uuid: "0ba70768088548b0babb46364900364f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } health_report { overall_health: HEALTHY } } }
Jun 20 15:16:19 dist-test-slave-7s64 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:19.593207 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.578884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37043 (local address 127.0.0.1:53058)
0620 15:16:19.579681 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:19.579695 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:19.579864 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:19.580174 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:19.580178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:19.580889 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:19.581542 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.581554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.582767 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.582774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.583821 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.583833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.584534 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.584540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.584707 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.585390 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:19.585411 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:19.587672 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:19.590740 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:19.590754 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:19.590762 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:19.591145 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:19.591491 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:19.591498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:19.591502 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:19.591634 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:19.592116 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:19.592125 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:19.592300 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 15:16:19.592469 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 15:16:19.592777 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":560,"threads_started":1}
I20260620 15:16:19.593379 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.578888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53058 (local address 127.0.97.126:37043)
0620 15:16:19.579564 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.579572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.579792 (+ 220us) server_negotiation.cc:407] Connection header received
0620 15:16:19.579984 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.579993 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.580050 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.580155 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:19.581690 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.582635 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.583958 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.584351 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.584488 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.587911 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:19.587915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:19.587919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:19.587944 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:19.590601 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:19.591271 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:19.591276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:19.591280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:19.591393 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:19.591732 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:19.591739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:19.591744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:19.591995 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:19.592139 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.592602 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.592902 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":492,"thread_start_us":327,"threads_started":1}
I20260620 15:16:19.599032 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:53058:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:16:19.600064 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:19.607015 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 39047bef2591489ea34141d6b24717d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a2318202763484b8c7cb32bc653a103]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:16:19.607035 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 39047bef2591489ea34141d6b24717d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a2318202763484b8c7cb32bc653a103]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:16:19.607173 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 39047bef2591489ea34141d6b24717d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3a2318202763484b8c7cb32bc653a103]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:16:19.607545 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39047bef2591489ea34141d6b24717d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:19.607556 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39047bef2591489ea34141d6b24717d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:19.607692 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39047bef2591489ea34141d6b24717d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:19.614790 2015 tablet_bootstrap.cc:492] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: Bootstrap starting.
I20260620 15:16:19.614802 2018 tablet_bootstrap.cc:492] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: Bootstrap starting.
I20260620 15:16:19.616063 2018 tablet_bootstrap.cc:654] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:19.616322 2015 tablet_bootstrap.cc:654] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:19.617643 2017 tablet_bootstrap.cc:492] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: Bootstrap starting.
I20260620 15:16:19.618427 2018 tablet_bootstrap.cc:492] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: No bootstrap required, opened a new log
I20260620 15:16:19.618530 2018 ts_tablet_manager.cc:1397] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 15:16:19.619149 2017 tablet_bootstrap.cc:654] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:19.619091 2018 raft_consensus.cc:348] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.619329 2018 raft_consensus.cc:374] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:19.619426 2018 raft_consensus.cc:729] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba70768088548b0babb46364900364f, State: Initialized, Role: FOLLOWER
I20260620 15:16:19.619666 2018 consensus_queue.cc:260] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.620141 2023 tablet_replica.cc:439] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } }
I20260620 15:16:19.620421 2023 tablet_replica.cc:442] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:19.623777 2018 ts_tablet_manager.cc:1428] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260620 15:16:19.627697 2015 tablet_bootstrap.cc:492] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: No bootstrap required, opened a new log
I20260620 15:16:19.627874 2015 ts_tablet_manager.cc:1397] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260620 15:16:19.628715 2017 tablet_bootstrap.cc:492] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: No bootstrap required, opened a new log
I20260620 15:16:19.628705 2015 raft_consensus.cc:348] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.628885 2015 raft_consensus.cc:374] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:19.628985 2015 raft_consensus.cc:729] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0431211c57bb490c8ac41ef5c106e546, State: Initialized, Role: FOLLOWER
I20260620 15:16:19.629019 2017 ts_tablet_manager.cc:1397] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260620 15:16:19.629225 2015 consensus_queue.cc:260] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.629659 2021 tablet_replica.cc:439] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } }
I20260620 15:16:19.629956 2021 tablet_replica.cc:442] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:19.629851 2017 raft_consensus.cc:348] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.630072 2017 raft_consensus.cc:374] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:19.630154 2017 raft_consensus.cc:729] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9029ef89b2c7430281faad74dc369ea5, State: Initialized, Role: FOLLOWER
I20260620 15:16:19.630375 2017 consensus_queue.cc:260] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.630848 2015 ts_tablet_manager.cc:1428] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 15:16:19.630820 2038 tablet_replica.cc:439] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } }
I20260620 15:16:19.631146 2038 tablet_replica.cc:442] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:19.632272 2017 ts_tablet_manager.cc:1428] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 15:16:19.714390 2038 raft_consensus.cc:482] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:19.714637 2038 raft_consensus.cc:504] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.715303 2038 leader_election.cc:288] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0431211c57bb490c8ac41ef5c106e546 (127.0.97.66:39201), 0ba70768088548b0babb46364900364f (127.0.97.67:37577)
I20260620 15:16:19.715839 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39047bef2591489ea34141d6b24717d5" candidate_uuid: "9029ef89b2c7430281faad74dc369ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431211c57bb490c8ac41ef5c106e546" is_pre_election: true
I20260620 15:16:19.715868 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39047bef2591489ea34141d6b24717d5" candidate_uuid: "9029ef89b2c7430281faad74dc369ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba70768088548b0babb46364900364f" is_pre_election: true
I20260620 15:16:19.716087 1807 raft_consensus.cc:2454] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9029ef89b2c7430281faad74dc369ea5 in term 0.
I20260620 15:16:19.716140 1951 raft_consensus.cc:2454] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9029ef89b2c7430281faad74dc369ea5 in term 0.
I20260620 15:16:19.716537 1599 leader_election.cc:302] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0431211c57bb490c8ac41ef5c106e546, 9029ef89b2c7430281faad74dc369ea5; no voters:
I20260620 15:16:19.716823 2038 raft_consensus.cc:2781] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:19.716957 2038 raft_consensus.cc:482] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:19.717077 2038 raft_consensus.cc:3037] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:19.719782 2038 raft_consensus.cc:504] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.720397 2038 leader_election.cc:288] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [CANDIDATE]: Term 1 election: Requested vote from peers 0431211c57bb490c8ac41ef5c106e546 (127.0.97.66:39201), 0ba70768088548b0babb46364900364f (127.0.97.67:37577)
I20260620 15:16:19.720865 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39047bef2591489ea34141d6b24717d5" candidate_uuid: "9029ef89b2c7430281faad74dc369ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0431211c57bb490c8ac41ef5c106e546"
I20260620 15:16:19.720935 1951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39047bef2591489ea34141d6b24717d5" candidate_uuid: "9029ef89b2c7430281faad74dc369ea5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ba70768088548b0babb46364900364f"
I20260620 15:16:19.721072 1807 raft_consensus.cc:3037] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:19.721133 1951 raft_consensus.cc:3037] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:19.722903 1807 raft_consensus.cc:2454] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9029ef89b2c7430281faad74dc369ea5 in term 1.
I20260620 15:16:19.723277 1599 leader_election.cc:302] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0431211c57bb490c8ac41ef5c106e546, 9029ef89b2c7430281faad74dc369ea5; no voters:
I20260620 15:16:19.723593 2038 raft_consensus.cc:2781] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:19.723748 2038 raft_consensus.cc:686] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [term 1 LEADER]: Becoming Leader. State: Replica: 9029ef89b2c7430281faad74dc369ea5, State: Running, Role: LEADER
I20260620 15:16:19.724056 1951 raft_consensus.cc:2454] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9029ef89b2c7430281faad74dc369ea5 in term 1.
I20260620 15:16:19.724026 2038 consensus_queue.cc:237] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } }
I20260620 15:16:19.726608 2022 tablet_replica.cc:439] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: TxnStatusTablet state changed. Reason: New leader 9029ef89b2c7430281faad74dc369ea5. Latest consensus state: current_term: 1 leader_uuid: "9029ef89b2c7430281faad74dc369ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } }
I20260620 15:16:19.726866 2022 tablet_replica.cc:442] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:16:19.727497 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:16:19.727710 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:16:19.727749 1511 catalog_manager.cc:5495] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9029ef89b2c7430281faad74dc369ea5 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "9029ef89b2c7430281faad74dc369ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } health_report { overall_health: UNKNOWN } } }
I20260620 15:16:19.767530 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.761427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:40867 (local address 127.0.0.1:59454)
0620 15:16:19.761695 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:19.761720 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:19.761890 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:19.762351 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:19.762356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:19.762370 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:16:19.762555 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.762567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.763687 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.763693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.765334 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.765351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.766375 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.766379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.766627 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.766699 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:16:19.767132 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:16:19.767144 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 15:16:19.767203 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 15:16:19.767311 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 15:16:19.767601 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.761538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59454 (local address 127.0.97.65:40867)
0620 15:16:19.761738 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.761743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.761787 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:16:19.762090 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.762093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.762156 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.762288 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:19.762751 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.763572 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.765536 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.766234 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.766371 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.766894 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:19.767025 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:19.767136 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.767230 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.767314 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 15:16:19.771113 1807 raft_consensus.cc:1261] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546 [term 1 FOLLOWER]: Refusing update from remote peer 9029ef89b2c7430281faad74dc369ea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:19.771174 1951 raft_consensus.cc:1261] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f [term 1 FOLLOWER]: Refusing update from remote peer 9029ef89b2c7430281faad74dc369ea5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:19.771710 2038 consensus_queue.cc:1035] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:19.772184 2022 consensus_queue.cc:1035] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:19.777434 2023 tablet_replica.cc:439] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: TxnStatusTablet state changed. Reason: New leader 9029ef89b2c7430281faad74dc369ea5. Latest consensus state: current_term: 1 leader_uuid: "9029ef89b2c7430281faad74dc369ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } }
I20260620 15:16:19.777711 2023 tablet_replica.cc:442] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:19.779080 2034 tablet_replica.cc:439] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9029ef89b2c7430281faad74dc369ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } }
I20260620 15:16:19.779388 2034 tablet_replica.cc:442] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:16:19.787569 2023 tablet_replica.cc:439] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9029ef89b2c7430281faad74dc369ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } }
I20260620 15:16:19.787946 2023 tablet_replica.cc:442] T 39047bef2591489ea34141d6b24717d5 P 0ba70768088548b0babb46364900364f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:19.791594 2021 tablet_replica.cc:439] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: TxnStatusTablet state changed. Reason: New leader 9029ef89b2c7430281faad74dc369ea5. Latest consensus state: current_term: 1 leader_uuid: "9029ef89b2c7430281faad74dc369ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } }
I20260620 15:16:19.791904 2021 tablet_replica.cc:442] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:19.791699 2038 tablet_replica.cc:439] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9029ef89b2c7430281faad74dc369ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } }
I20260620 15:16:19.792016 2038 tablet_replica.cc:442] T 39047bef2591489ea34141d6b24717d5 P 9029ef89b2c7430281faad74dc369ea5: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:16:19.794407 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298943302736793600 to 7298943302552092672. Current Snapshot: MvccSnapshot[applied={T|T < 7298943302736793600}]
I20260620 15:16:19.794903 2021 tablet_replica.cc:439] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9029ef89b2c7430281faad74dc369ea5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 } } peers { permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 } } peers { permanent_uuid: "0ba70768088548b0babb46364900364f" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37577 } } }
I20260620 15:16:19.795132 2021 tablet_replica.cc:442] T 39047bef2591489ea34141d6b24717d5 P 0431211c57bb490c8ac41ef5c106e546: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:19.795524 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.787557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.0.97.67:37577)
0620 15:16:19.788287 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.788296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.788323 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:16:19.788440 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.788447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.788590 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.789323 (+ 733us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:19.789480 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.790700 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.792584 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.793227 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.793727 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.793857 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:19.794602 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:19.795176 (+ 574us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.795227 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.795288 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":420,"threads_started":1}
I20260620 15:16:19.812592 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.806204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:40867 (local address 127.0.97.67:58483)
0620 15:16:19.806881 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:19.806897 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:19.807092 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:19.807441 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:19.807444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:19.807454 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:16:19.807704 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.807710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.809222 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.809226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.810261 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.810272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.811372 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.811376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.811484 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.811535 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:16:19.811956 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:16:19.811966 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:16:19.812079 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 15:16:19.812208 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":352,"threads_started":1}
I20260620 15:16:19.813241 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.806212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:58483 (local address 127.0.97.65:40867)
0620 15:16:19.806427 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.806434 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.807060 (+ 626us) server_negotiation.cc:407] Connection header received
0620 15:16:19.807223 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.807231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.807323 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.807436 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:19.807889 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.809097 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.810416 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.811210 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.811363 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.811705 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:19.811836 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:19.811950 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.812876 (+ 926us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.812954 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:16:19.827255 1806 raft_consensus.cc:1261] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546 [term 1 FOLLOWER]: Refusing update from remote peer 0ba70768088548b0babb46364900364f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:19.827802 1668 raft_consensus.cc:1261] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5 [term 1 FOLLOWER]: Refusing update from remote peer 0ba70768088548b0babb46364900364f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:19.827982 2048 consensus_queue.cc:1035] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0431211c57bb490c8ac41ef5c106e546" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:19.828598 2048 consensus_queue.cc:1035] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9029ef89b2c7430281faad74dc369ea5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:19.842980 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298943302966902784 to 7298943301848264704. Current Snapshot: MvccSnapshot[applied={T|T < 7298943302966902784}]
I20260620 15:16:19.886812 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.879032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37577 (local address 127.0.97.65:43431)
0620 15:16:19.879799 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:19.879819 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:19.880021 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:19.880669 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:19.880674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:19.880686 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:16:19.881612 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.881625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.883135 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.883140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.884441 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:19.884452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.885534 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.885542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:19.885737 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.885820 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:16:19.886357 (+ 537us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:16:19.886366 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 15:16:19.886430 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 15:16:19.886519 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":395,"threads_started":1}
I20260620 15:16:19.886847 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.879035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:43431 (local address 127.0.97.67:37577)
0620 15:16:19.879188 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.879194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.879915 (+ 721us) server_negotiation.cc:407] Connection header received
0620 15:16:19.880329 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.880335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.880408 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.880461 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:19.881799 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.882914 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.884619 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.885347 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.885495 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.886120 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:19.886252 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:19.886359 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.886458 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.886542 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:16:19.925397 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:19.925655 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:19.929148 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e407b154ee3748d9b9d7f35efef530f5: Sending DeleteTablet for 3 replicas of tablet 59425e8d0f454eb3a6269dff69780154
I20260620 15:16:19.930263 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 59425e8d0f454eb3a6269dff69780154 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:19 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:42822
I20260620 15:16:19.931547 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 59425e8d0f454eb3a6269dff69780154 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:19 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:41692
I20260620 15:16:19.934010 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 59425e8d0f454eb3a6269dff69780154 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:19 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:35234
I20260620 15:16:19.934275 2042 tablet_replica.cc:330] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5: stopping tablet replica
I20260620 15:16:19.934471 2042 raft_consensus.cc:2229] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:19.934654 2042 raft_consensus.cc:2258] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:19.935516 2078 tablet_replica.cc:330] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f: stopping tablet replica
I20260620 15:16:19.935810 2078 raft_consensus.cc:2229] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:16:19.936012 2042 ts_tablet_manager.cc:1905] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:19.936507 2078 raft_consensus.cc:2258] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:19.941802 2078 ts_tablet_manager.cc:1905] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:19.944983 2042 ts_tablet_manager.cc:1918] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:16:19.948043 2042 log.cc:1196] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/59425e8d0f454eb3a6269dff69780154
I20260620 15:16:19.948756 2042 ts_tablet_manager.cc:1939] T 59425e8d0f454eb3a6269dff69780154 P 9029ef89b2c7430281faad74dc369ea5: Deleting consensus metadata
I20260620 15:16:19.950786 1498 catalog_manager.cc:4841] TS 9029ef89b2c7430281faad74dc369ea5 (127.0.97.65:40867): tablet 59425e8d0f454eb3a6269dff69780154 (table test-table [id=175b1ac3371d489c98007db0194a58b4]) successfully deleted
I20260620 15:16:19.954095 2076 tablet_replica.cc:330] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546: stopping tablet replica
I20260620 15:16:19.954284 2076 raft_consensus.cc:2229] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:19.954561 2076 raft_consensus.cc:2258] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:19.956439 2076 ts_tablet_manager.cc:1905] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:19.964644 2078 ts_tablet_manager.cc:1918] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:16:19.964830 2078 log.cc:1196] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/59425e8d0f454eb3a6269dff69780154
I20260620 15:16:19.965289 2078 ts_tablet_manager.cc:1939] T 59425e8d0f454eb3a6269dff69780154 P 0ba70768088548b0babb46364900364f: Deleting consensus metadata
I20260620 15:16:19.966396 1498 catalog_manager.cc:4841] TS 0ba70768088548b0babb46364900364f (127.0.97.67:37577): tablet 59425e8d0f454eb3a6269dff69780154 (table test-table [id=175b1ac3371d489c98007db0194a58b4]) successfully deleted
I20260620 15:16:19.966542 2076 ts_tablet_manager.cc:1918] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:16:19.966725 2076 log.cc:1196] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/59425e8d0f454eb3a6269dff69780154
I20260620 15:16:19.967348 2076 ts_tablet_manager.cc:1939] T 59425e8d0f454eb3a6269dff69780154 P 0431211c57bb490c8ac41ef5c106e546: Deleting consensus metadata
I20260620 15:16:19.968533 1496 catalog_manager.cc:4841] TS 0431211c57bb490c8ac41ef5c106e546 (127.0.97.66:39201): tablet 59425e8d0f454eb3a6269dff69780154 (table test-table [id=175b1ac3371d489c98007db0194a58b4]) successfully deleted
Jun 20 15:16:19 dist-test-slave-7s64 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:19.982236 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.954535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.0.97.65:40867)
0620 15:16:19.954746 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.954750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.964760 (+ 10010us) server_negotiation.cc:407] Connection header received
0620 15:16:19.965290 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.965296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.965386 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.965461 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:19.967811 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.969012 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.970618 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.971074 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.971218 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.975847 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:19.975860 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:19.975875 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:19.975906 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:19.978877 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:19.979599 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:19.979608 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:19.979619 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:19.979733 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:19.980084 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:19.980091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:19.980095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:19.980441 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:19.980840 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.981524 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.981794 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 15:16:19.983023 1711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59462
I20260620 15:16:19.999771 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:19.986582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.0.97.126:37043)
0620 15:16:19.986829 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:19.986836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:19.987062 (+ 226us) server_negotiation.cc:407] Connection header received
0620 15:16:19.987241 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:19.987246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:19.987343 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:19.987433 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:19.989129 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.989852 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.991046 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:19.991469 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:19.991594 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:19.994175 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:19.994180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:19.994185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:19.994223 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:19.996842 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:19.997459 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:19.997467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:19.997470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:19.997596 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:19.997942 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:19.997948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:19.997952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:19.998225 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:19.998474 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:19.999183 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 15:16:19.999428 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 15:16:20.000342 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53062
I20260620 15:16:20.002802 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 1582
I20260620 15:16:20.017489 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:20.003881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.0.97.65:40867)
0620 15:16:20.004055 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:20.004061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:20.005501 (+ 1440us) server_negotiation.cc:407] Connection header received
0620 15:16:20.005641 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:20.005646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:20.005723 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:20.005848 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:20.007033 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:20.007816 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:20.009020 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:20.009495 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:20.009632 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:20.011665 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:20.011671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:20.011676 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:20.011717 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:20.014869 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:20.015468 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:20.015474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:20.015478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:20.015624 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:20.015987 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:20.015992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:20.015995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:20.016228 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:20.016528 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:20.016792 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 15:16:20.017002 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:16:20.018064 1711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59470
W20260620 15:16:20.018566 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:16:20.018759 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 1582
I20260620 15:16:20.042147 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 1722
Jun 20 15:16:20 dist-test-slave-7s64 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:16:20.058580 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:20.042884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.0.97.66:39201)
0620 15:16:20.043562 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:20.043567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:20.043583 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:16:20.043634 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:20.043637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:20.043704 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:20.043796 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:20.045256 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:20.045958 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:20.047740 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:20.048121 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:20.048209 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:20.052222 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:20.052239 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:20.052259 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:20.052298 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:20.055917 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:20.056665 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:20.056675 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:20.056684 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:20.056809 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:20.057171 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:20.057175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:20.057177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:20.057478 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:20.057776 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:20.058068 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 15:16:20.058218 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":304,"threads_started":1}
W20260620 15:16:20.059248 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40420
W20260620 15:16:20.059887 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:16:20.060037 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 1722
I20260620 15:16:20.078409 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 1861
Jun 20 15:16:20 dist-test-slave-7s64 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968578, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:16:20.094520 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:20.079244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.0.97.67:37577)
0620 15:16:20.079469 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:20.079475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:20.079521 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:16:20.079588 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:20.079593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:20.079668 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:20.079766 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:20.080931 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:20.082154 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:20.083588 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:20.084025 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:20.084157 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:20.088085 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:20.088100 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:20.088121 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:20.088167 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:20.091937 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:20.092541 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:20.092550 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:20.092554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:20.092726 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:20.093059 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:20.093064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:20.093067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:20.093493 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:20.093800 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:20.093946 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 15:16:20.094142 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 15:16:20.095258 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44164
W20260620 15:16:20.095870 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:16:20.096030 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 1861
I20260620 15:16:20.113734 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 1479
I20260620 15:16:20.114735 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:20.226326 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 1479
2026-06-20T15:16:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4637 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:20 dist-test-slave-7s64 krb5kdc[2087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:20 dist-test-slave-7s64 krb5kdc[2087](info): set up 2 sockets
Jun 20 15:16:20 dist-test-slave-7s64 krb5kdc[2087](info): commencing operation
krb5kdc: starting...
W20260620 15:16:21.300474 389 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:21 dist-test-slave-7s64 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:16:21.476615 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43667
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:21.780539 2100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:21.780819 2100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:21.780969 2100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:21.787707 2100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:21.787794 2100 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:16:21.787866 2100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:21.787958 2100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:21.788062 2100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:21.788163 2100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:21.797556 2100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43667
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:21.798689 2100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:21.799969 2100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:21.810942 2105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:21.811007 2106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:21.811029 2108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:21.812044 2100 server_base.cc:1034] running on GCE node
I20260620 15:16:21.812755 2100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:21.813977 2100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:21.815198 2100 hybrid_clock.cc:648] HybridClock initialized: now 1781968581815116 us; error 51 us; skew 500 ppm
I20260620 15:16:21.815521 2100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:21 dist-test-slave-7s64 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:21.818770 2100 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:16:21.820065 2100 webserver.cc:460] Webserver started at http://127.0.97.126:46223/ using document root <none> and password file <none>
I20260620 15:16:21.820817 2100 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:21.820940 2100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:21.821235 2100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:21.823920 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "8df8c35c3be8406782dd524de95e6320"
format_stamp: "Formatted at 2026-06-20 15:16:21 on dist-test-slave-7s64"
server_key: "2cbe6aea27b9985749f9693406887474"
server_key_iv: "b5cd5d375e982aa87a234ac31afa393c"
server_key_version: "encryptionkey@0"
I20260620 15:16:21.824716 2100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "8df8c35c3be8406782dd524de95e6320"
format_stamp: "Formatted at 2026-06-20 15:16:21 on dist-test-slave-7s64"
server_key: "2cbe6aea27b9985749f9693406887474"
server_key_iv: "b5cd5d375e982aa87a234ac31afa393c"
server_key_version: "encryptionkey@0"
I20260620 15:16:21.829418 2100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:16:21.832588 2115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:21.833760 2100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:16:21.833920 2100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "8df8c35c3be8406782dd524de95e6320"
format_stamp: "Formatted at 2026-06-20 15:16:21 on dist-test-slave-7s64"
server_key: "2cbe6aea27b9985749f9693406887474"
server_key_iv: "b5cd5d375e982aa87a234ac31afa393c"
server_key_version: "encryptionkey@0"
I20260620 15:16:21.834117 2100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:21.855480 2100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:21.859249 2100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:21.859519 2100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:21.880052 2100 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43667
I20260620 15:16:21.880067 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43667 every 8 connection(s)
I20260620 15:16:21.881381 2100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:21.884874 2177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:21.891644 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 2100
I20260620 15:16:21.891875 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:21.892474 389 external_mini_cluster.cc:1428] Setting key 069440c00d93b27d63d3431e2ca25e5e
I20260620 15:16:21.894598 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320: Bootstrap starting.
I20260620 15:16:21.898591 2177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:21.899677 2177 log.cc:824] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:21.902585 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320: No bootstrap required, opened a new log
Jun 20 15:16:21 dist-test-slave-7s64 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:21.911484 2177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df8c35c3be8406782dd524de95e6320" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43667 } }
I20260620 15:16:21.911952 2177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:21.912058 2177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df8c35c3be8406782dd524de95e6320, State: Initialized, Role: FOLLOWER
I20260620 15:16:21.912700 2177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df8c35c3be8406782dd524de95e6320" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43667 } }
I20260620 15:16:21.913002 2177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:21.913192 2177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:21.913369 2177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:21.916358 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:21.894486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.0.97.126:43667)
0620 15:16:21.895321 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:21.895336 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:21.895383 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:16:21.897808 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:21.897835 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:21.898286 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:21.898702 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:21.900041 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:21.902010 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:21.903271 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:21.904251 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:21.904443 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:21.908093 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:21.908105 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:21.908127 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:21.908170 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:21.912052 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:21.912771 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:21.912778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:21.912785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:21.912965 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:21.913372 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:21.913379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:21.913385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:21.914057 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:21.914333 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:21.914621 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 15:16:21.915058 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":362,"threads_started":1}
I20260620 15:16:21.916602 2177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df8c35c3be8406782dd524de95e6320" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43667 } }
I20260620 15:16:21.917137 2177 leader_election.cc:302] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8df8c35c3be8406782dd524de95e6320; no voters:
I20260620 15:16:21.917726 2177 leader_election.cc:288] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:21.917900 2182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:21.918565 2182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [term 1 LEADER]: Becoming Leader. State: Replica: 8df8c35c3be8406782dd524de95e6320, State: Running, Role: LEADER
I20260620 15:16:21.919165 2182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df8c35c3be8406782dd524de95e6320" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43667 } }
I20260620 15:16:21.920836 2177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:21.921983 2184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8df8c35c3be8406782dd524de95e6320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df8c35c3be8406782dd524de95e6320" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43667 } } }
I20260620 15:16:21.922380 2184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:21.923029 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8df8c35c3be8406782dd524de95e6320. Latest consensus state: current_term: 1 leader_uuid: "8df8c35c3be8406782dd524de95e6320" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8df8c35c3be8406782dd524de95e6320" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43667 } } }
I20260620 15:16:21.923390 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:21.923915 2188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:21.928164 2188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:21.939412 2188 catalog_manager.cc:1347] Generated new cluster ID: 3f9f33fc3165475b98d08d5826ea3e80
I20260620 15:16:21.939538 2188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:21.951460 2188 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:21.952685 2188 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:21.962765 2188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320: Generated new TSK 0
I20260620 15:16:21.963485 2188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:16:22.006078 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43667
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:22.320860 2203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:22.321144 2203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:22.321358 2203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:22.328403 2203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:22.328504 2203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:22.328626 2203 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:16:22.328684 2203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:16:22.328751 2203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:22.338224 2203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:43667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:22.339322 2203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:22.340574 2203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:22.352887 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:22.352800 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:22.352856 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:22.353612 2203 server_base.cc:1034] running on GCE node
I20260620 15:16:22.354233 2203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:22.355111 2203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:22.356328 2203 hybrid_clock.cc:648] HybridClock initialized: now 1781968582356292 us; error 54 us; skew 500 ppm
I20260620 15:16:22.356664 2203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:22 dist-test-slave-7s64 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:22.359668 2203 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:16:22.361025 2203 webserver.cc:460] Webserver started at http://127.0.97.65:44235/ using document root <none> and password file <none>
I20260620 15:16:22.361647 2203 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:22.361783 2203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:22.362066 2203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:22.364746 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "12f2fec3be6c47ea821688657a1cac4a"
format_stamp: "Formatted at 2026-06-20 15:16:22 on dist-test-slave-7s64"
server_key: "f90f47a1e4c8c7d14bc13f340c3eb0d4"
server_key_iv: "a1268784f9d232d4c424a35fd31712e0"
server_key_version: "encryptionkey@0"
I20260620 15:16:22.365485 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "12f2fec3be6c47ea821688657a1cac4a"
format_stamp: "Formatted at 2026-06-20 15:16:22 on dist-test-slave-7s64"
server_key: "f90f47a1e4c8c7d14bc13f340c3eb0d4"
server_key_iv: "a1268784f9d232d4c424a35fd31712e0"
server_key_version: "encryptionkey@0"
I20260620 15:16:22.370113 2203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:16:22.373276 2218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:22.374459 2203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:22.374615 2203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "12f2fec3be6c47ea821688657a1cac4a"
format_stamp: "Formatted at 2026-06-20 15:16:22 on dist-test-slave-7s64"
server_key: "f90f47a1e4c8c7d14bc13f340c3eb0d4"
server_key_iv: "a1268784f9d232d4c424a35fd31712e0"
server_key_version: "encryptionkey@0"
I20260620 15:16:22.374794 2203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:22.403456 2203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:22.407207 2203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:22.407486 2203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:22.410003 2203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:22.410118 2203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:22.410233 2203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:22.410344 2203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:16:22 dist-test-slave-7s64 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:22.437176 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:22.411099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43667 (local address 127.0.97.65:45197)
0620 15:16:22.411966 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:22.412010 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:22.413976 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:22.414796 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:22.414810 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:22.415615 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:22.416678 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:22.416701 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.418160 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.418172 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:22.421071 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:22.421090 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.422494 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.422504 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:22.423073 (+ 569us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:22.423857 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:22.423916 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:22.426789 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:22.431472 (+ 4683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:22.431478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:22.431489 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:22.431979 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:22.432465 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:22.432473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:22.432477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:22.432975 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:22.433722 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:22.433734 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:22.434294 (+ 560us) client_negotiation.cc:769] Sending connection context
0620 15:16:22.434675 (+ 381us) client_negotiation.cc:240] Negotiation successful
0620 15:16:22.435287 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":364,"threads_started":1}
I20260620 15:16:22.437950 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:22.410514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:45197 (local address 127.0.97.126:43667)
0620 15:16:22.410707 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:22.410713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:22.412140 (+ 1427us) server_negotiation.cc:407] Connection header received
0620 15:16:22.414273 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:22.414281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:22.414378 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:22.414510 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:22.416930 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.417974 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.421291 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.422065 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.422227 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:22.427042 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:22.427051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:22.427059 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:22.427109 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:22.431289 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:22.432137 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:22.432143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:22.432148 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:22.432325 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:22.433202 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:22.433210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:22.433214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:22.433570 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:22.433749 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:22.437208 (+ 3459us) server_negotiation.cc:299] Negotiation successful
0620 15:16:22.437467 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:16:22.464143 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:22.447253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57491 (local address 127.0.97.126:43667)
0620 15:16:22.447474 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:22.447482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:22.448173 (+ 691us) server_negotiation.cc:407] Connection header received
0620 15:16:22.448460 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:22.448467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:22.448586 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:22.448661 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:22.450352 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.451062 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.453822 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.454577 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.454684 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:22.457747 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:22.457766 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:22.457774 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:22.457819 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:22.461168 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:22.462086 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:22.462094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:22.462100 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:22.462274 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:22.462769 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:22.462772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:22.462775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:22.463135 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:22.463263 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:22.463773 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 15:16:22.463924 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:16:22.464774 2279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:22.447218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43667 (local address 127.0.97.65:57491)
0620 15:16:22.448069 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:22.448094 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:22.448291 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:22.448909 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:22.448916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:22.449650 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:22.450152 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:22.450179 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.451237 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.451245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:22.453600 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:22.453621 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.455032 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.455042 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:22.455303 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:22.456100 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:22.456130 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:22.457516 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:22.461314 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:22.461319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:22.461323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:22.461939 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:22.462460 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:22.462468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:22.462475 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:22.462651 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:22.463252 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:22.463256 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:22.463503 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 15:16:22.463682 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 15:16:22.464092 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":461,"threads_started":1}
I20260620 15:16:22.471884 2203 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42711
I20260620 15:16:22.471890 2337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42711 every 8 connection(s)
I20260620 15:16:22.473078 2203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:22.474965 2338 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43667
I20260620 15:16:22.475169 2338 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:22.475770 2338 heartbeater.cc:511] Master 127.0.97.126:43667 requested a full tablet report, sending...
I20260620 15:16:22.477537 2132 ts_manager.cc:194] Registered new tserver with Master: 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711)
I20260620 15:16:22.478961 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:45197
I20260620 15:16:22.482218 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 2203
I20260620 15:16:22.482412 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:22.482996 389 external_mini_cluster.cc:1428] Setting key d3256d8bcee2edfb61eb151e26149afe
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:16:22.521270 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43667
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:22.830396 2342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:22.830708 2342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:22.830881 2342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:22.837797 2342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:22.837926 2342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:22.838074 2342 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:16:22.838162 2342 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:16:22.838300 2342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:22.847620 2342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:43667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:22.848794 2342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:22.850013 2342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:22.862073 2350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:22.862141 2347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:22.862140 2348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:22.862884 2342 server_base.cc:1034] running on GCE node
I20260620 15:16:22.863507 2342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:22.864403 2342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:22.865644 2342 hybrid_clock.cc:648] HybridClock initialized: now 1781968582865572 us; error 47 us; skew 500 ppm
I20260620 15:16:22.866045 2342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:22 dist-test-slave-7s64 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:22.869099 2342 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:16:22.870412 2342 webserver.cc:460] Webserver started at http://127.0.97.66:38083/ using document root <none> and password file <none>
I20260620 15:16:22.871022 2342 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:22.871145 2342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:22.871439 2342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:22.874132 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "d912d63708b14563b77d0f1990dc681e"
format_stamp: "Formatted at 2026-06-20 15:16:22 on dist-test-slave-7s64"
server_key: "58215e3b5ac6a8e4c6820ae90593e006"
server_key_iv: "5f61500414b58a85649a4714cf386348"
server_key_version: "encryptionkey@0"
I20260620 15:16:22.874861 2342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "d912d63708b14563b77d0f1990dc681e"
format_stamp: "Formatted at 2026-06-20 15:16:22 on dist-test-slave-7s64"
server_key: "58215e3b5ac6a8e4c6820ae90593e006"
server_key_iv: "5f61500414b58a85649a4714cf386348"
server_key_version: "encryptionkey@0"
I20260620 15:16:22.879971 2342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 15:16:22.883294 2357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:22.884764 2342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:22.884933 2342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "d912d63708b14563b77d0f1990dc681e"
format_stamp: "Formatted at 2026-06-20 15:16:22 on dist-test-slave-7s64"
server_key: "58215e3b5ac6a8e4c6820ae90593e006"
server_key_iv: "5f61500414b58a85649a4714cf386348"
server_key_version: "encryptionkey@0"
I20260620 15:16:22.885116 2342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:22.910192 2342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:22.913968 2342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:22.914253 2342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:22.916678 2342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:22.916821 2342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:22.916930 2342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:22.917003 2342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:16:22 dist-test-slave-7s64 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:22.939510 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:22.917347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:37095 (local address 127.0.97.126:43667)
0620 15:16:22.917530 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:22.917536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:22.919003 (+ 1467us) server_negotiation.cc:407] Connection header received
0620 15:16:22.921134 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:22.921139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:22.921225 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:22.921329 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:22.923204 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.924254 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.926742 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.927339 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.927462 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:22.932154 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:22.932162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:22.932169 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:22.932219 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:22.935513 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:22.936522 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:22.936528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:22.936533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:22.936652 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:22.937248 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:22.937256 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:22.937262 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:22.937723 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:22.937891 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:22.938727 (+ 836us) server_negotiation.cc:299] Negotiation successful
0620 15:16:22.939013 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:16:22.940157 2366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:22.918016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43667 (local address 127.0.97.66:37095)
0620 15:16:22.918859 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:22.918893 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:22.920876 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:22.921570 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:22.921584 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:22.922261 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:22.923029 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:22.923057 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.924397 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.924404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:22.926544 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:22.926562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.927577 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.927589 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:22.927909 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:22.928638 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:22.928680 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:22.931889 (+ 3209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:22.935637 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:22.935645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:22.935658 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:22.936376 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:22.936793 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:22.936801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:22.936805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:22.937098 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:22.937881 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:22.937888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:22.938341 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 15:16:22.938545 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 15:16:22.938942 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":405,"threads_started":1}
I20260620 15:16:22.960208 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:22.945676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41595 (local address 127.0.97.126:43667)
0620 15:16:22.945834 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:22.945839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:22.946766 (+ 927us) server_negotiation.cc:407] Connection header received
0620 15:16:22.946945 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:22.946948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:22.946998 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:22.947102 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:22.948642 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.949696 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.951795 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.952339 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.952478 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:22.954629 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:22.954634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:22.954638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:22.954674 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:22.957386 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:22.958315 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:22.958322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:22.958329 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:22.958467 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:22.958805 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:22.958812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:22.958816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:22.959085 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:22.959232 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:22.959629 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 15:16:22.959844 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:16:22.960327 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:22.945747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43667 (local address 127.0.97.66:41595)
0620 15:16:22.946644 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:22.946669 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:22.946826 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:22.947130 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:22.947135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:22.947764 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:22.948479 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:22.948518 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.949819 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.949825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:22.951583 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:22.951606 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:22.952616 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:22.952624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:22.952772 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:22.953444 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:22.953472 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:22.954459 (+ 987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:22.957560 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:22.957568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:22.957573 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:22.958070 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:22.958562 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:22.958568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:22.958572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:22.958693 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:22.959199 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:22.959212 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:22.959401 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 15:16:22.959549 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 15:16:22.959846 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":436,"threads_started":1}
I20260620 15:16:22.967368 2342 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:33051
I20260620 15:16:22.967442 2476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:33051 every 8 connection(s)
I20260620 15:16:22.968675 2342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:22.970546 2477 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43667
I20260620 15:16:22.970765 2477 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:22.971333 2477 heartbeater.cc:511] Master 127.0.97.126:43667 requested a full tablet report, sending...
I20260620 15:16:22.972476 2132 ts_manager.cc:194] Registered new tserver with Master: d912d63708b14563b77d0f1990dc681e (127.0.97.66:33051)
I20260620 15:16:22.973255 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:37095
I20260620 15:16:22.978015 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 2342
I20260620 15:16:22.978222 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:22.978649 389 external_mini_cluster.cc:1428] Setting key 720b741170ec82ceeca820c32fb9ca2c
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:16:23.015453 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43667
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:23.326211 2481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:23.326464 2481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:23.326625 2481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:23.333746 2481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:23.333835 2481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:23.333987 2481 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:16:23.334062 2481 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:16:23.334192 2481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:23.343670 2481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:43667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:23.344970 2481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:23.346177 2481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:23.358592 2486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:23.358666 2489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:23.358820 2487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:23.359726 2481 server_base.cc:1034] running on GCE node
I20260620 15:16:23.360275 2481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:23.361230 2481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:23.362466 2481 hybrid_clock.cc:648] HybridClock initialized: now 1781968583362437 us; error 48 us; skew 500 ppm
I20260620 15:16:23.362754 2481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:23 dist-test-slave-7s64 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:23.370321 2481 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:16:23.371693 2481 webserver.cc:460] Webserver started at http://127.0.97.67:38807/ using document root <none> and password file <none>
I20260620 15:16:23.372327 2481 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:23.372467 2481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:23.372804 2481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:23.375496 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "eb720c0dd61f4e69ab048c03575c3b73"
format_stamp: "Formatted at 2026-06-20 15:16:23 on dist-test-slave-7s64"
server_key: "d95d566b4367a4b5784dc44948e68c67"
server_key_iv: "cd49d12fd1d9122731b045b8a3d4b2a5"
server_key_version: "encryptionkey@0"
I20260620 15:16:23.376233 2481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "eb720c0dd61f4e69ab048c03575c3b73"
format_stamp: "Formatted at 2026-06-20 15:16:23 on dist-test-slave-7s64"
server_key: "d95d566b4367a4b5784dc44948e68c67"
server_key_iv: "cd49d12fd1d9122731b045b8a3d4b2a5"
server_key_version: "encryptionkey@0"
I20260620 15:16:23.381433 2481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:16:23.384737 2496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:23.385980 2481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:16:23.386157 2481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "eb720c0dd61f4e69ab048c03575c3b73"
format_stamp: "Formatted at 2026-06-20 15:16:23 on dist-test-slave-7s64"
server_key: "d95d566b4367a4b5784dc44948e68c67"
server_key_iv: "cd49d12fd1d9122731b045b8a3d4b2a5"
server_key_version: "encryptionkey@0"
I20260620 15:16:23.386338 2481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:23.410964 2481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:23.414673 2481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:23.414953 2481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:23.417522 2481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:23.417658 2481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:23.417789 2481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:23.417888 2481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:16:23 dist-test-slave-7s64 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:23.444456 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.418627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43667 (local address 127.0.97.67:41981)
0620 15:16:23.419532 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.419586 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.421796 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.422635 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.422643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.423429 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:23.424364 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.424398 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.426278 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.426288 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.429430 (+ 3142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.429449 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.430382 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.430390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.430610 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.432166 (+ 1556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:23.432193 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:23.435445 (+ 3252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:23.439582 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:23.439588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:23.439598 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:23.439986 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:23.440530 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:23.440536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:23.440542 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:23.440823 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:23.441580 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:23.441587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:23.441937 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.442302 (+ 365us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.442845 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":419,"threads_started":1}
I20260620 15:16:23.445137 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.418009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:41981 (local address 127.0.97.126:43667)
0620 15:16:23.418216 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.418224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.419730 (+ 1506us) server_negotiation.cc:407] Connection header received
0620 15:16:23.422111 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.422122 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.422255 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.422380 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:23.424600 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.425996 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.429585 (+ 3589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.430104 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.430265 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.435661 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:23.435665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:23.435668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:23.435698 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:23.439402 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:23.440127 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:23.440135 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:23.440141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:23.440391 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:23.441006 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:23.441015 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:23.441021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:23.441443 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:23.441610 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.444480 (+ 2870us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.444797 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:16:23.464273 2481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:40709
I20260620 15:16:23.464323 2615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:40709 every 8 connection(s)
I20260620 15:16:23.465587 2481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:23.466261 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.452278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55789 (local address 127.0.97.126:43667)
0620 15:16:23.452472 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.452479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.452900 (+ 421us) server_negotiation.cc:407] Connection header received
0620 15:16:23.453080 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.453086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.453170 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.453279 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:23.454753 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.455386 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.457565 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.458181 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.458318 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.460430 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:23.460439 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:23.460445 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:23.460519 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:23.463726 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:23.464396 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:23.464404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:23.464410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:23.464593 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:23.464934 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:23.464941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:23.464945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:23.465172 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:23.465317 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.465689 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.465901 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:16:23.466400 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.452021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43667 (local address 127.0.97.67:55789)
0620 15:16:23.452792 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.452816 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.452957 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.453456 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.453462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.454107 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:23.454574 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.454590 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.455548 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.455551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.457409 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.457421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.458360 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.458372 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.458562 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.459151 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:23.459178 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:23.460262 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:23.463858 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:23.463864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:23.463868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:23.464277 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:23.464700 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:23.464706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:23.464709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:23.464831 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:23.465295 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:23.465300 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:23.465467 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.465610 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.465913 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":401,"threads_started":1}
I20260620 15:16:23.467926 2616 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43667
I20260620 15:16:23.468438 2616 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:23.469213 2616 heartbeater.cc:511] Master 127.0.97.126:43667 requested a full tablet report, sending...
I20260620 15:16:23.470228 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 2481
I20260620 15:16:23.470427 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:23.470717 2132 ts_manager.cc:194] Registered new tserver with Master: eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709)
I20260620 15:16:23.470799 389 external_mini_cluster.cc:1428] Setting key f3777c41694d8e9f5267ee6362cca64d
I20260620 15:16:23.471730 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:41981
I20260620 15:16:23.473778 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:16:23.481513 2338 heartbeater.cc:503] Master 127.0.97.126:43667 was elected leader, sending a full tablet report...
Jun 20 15:16:23 dist-test-slave-7s64 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:16:23 dist-test-slave-7s64 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:23.519497 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.507015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.0.97.126:43667)
0620 15:16:23.507177 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.507181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.507525 (+ 344us) server_negotiation.cc:407] Connection header received
0620 15:16:23.507668 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.507672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.507730 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.507810 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:23.509097 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.509837 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.510952 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.511354 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.511441 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.514923 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:23.514927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:23.514931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:23.514962 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:23.517445 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:23.517959 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:23.517963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:23.517965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:23.518079 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:23.518360 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:23.518363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:23.518365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:23.518585 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:23.518740 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.519093 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.519239 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:16:23.523510 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:16:23.530166 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:23.564148 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.542382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33051 (local address 127.0.0.1:52356)
0620 15:16:23.556330 (+ 13948us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.556351 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.556511 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.557073 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.557077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.557105 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:23.557641 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.557658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.559524 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.559529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.561437 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.561448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.563115 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.563122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.563306 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.563378 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.563511 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.563696 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13780,"mutex_wait_us":93,"thread_start_us":13633,"threads_started":1}
I20260620 15:16:23.564515 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.541596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.0.97.67:40709)
0620 15:16:23.556799 (+ 15203us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.556806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.556834 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:16:23.556918 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.556926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.557154 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.557970 (+ 816us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:23.558298 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.559910 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.562199 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.563244 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.563698 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.563907 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.563975 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.564147 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15053,"thread_start_us":14839,"threads_started":1}
I20260620 15:16:23.564894 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.541324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40709 (local address 127.0.0.1:55648)
0620 15:16:23.548955 (+ 7631us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.548995 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.549146 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.557365 (+ 8219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.557373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.557400 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:23.557756 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.557788 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.560053 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.560058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.562062 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.562072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.563476 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.563485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.563710 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.563752 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.563828 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.563929 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7412,"mutex_wait_us":47,"thread_start_us":7296,"threads_started":1}
I20260620 15:16:23.565176 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.542652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.0.97.66:33051)
0620 15:16:23.553781 (+ 11129us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.553795 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.556645 (+ 2850us) server_negotiation.cc:407] Connection header received
0620 15:16:23.556738 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.556742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.556903 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.557086 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:23.557864 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.559360 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.561582 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.562922 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.563054 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.563261 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.564600 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.564884 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10997,"thread_start_us":11124,"threads_started":1}
I20260620 15:16:23.565312 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.542283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.0.97.65:42711)
0620 15:16:23.555897 (+ 13614us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.555905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.555932 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:16:23.556023 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.556032 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.556285 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.556512 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:23.557374 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.558510 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.561844 (+ 3334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.563174 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.563314 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.563516 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.564854 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.565010 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13456,"thread_start_us":13330,"threads_started":1}
I20260620 15:16:23.565431 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.541720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42711 (local address 127.0.0.1:40662)
0620 15:16:23.549626 (+ 7906us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.549646 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.549796 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.556775 (+ 6979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.556810 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.556851 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:23.557219 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.557233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.558649 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.558654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.561674 (+ 3020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.561693 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.563431 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.563437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.563620 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.563691 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.563765 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.563903 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7750,"mutex_wait_us":2,"thread_start_us":7593,"threads_started":1}
I20260620 15:16:23.568480 2268 tablet_service.cc:1467] Processing CreateTablet for tablet ebe1b842a5354c94b5c486f72e108e28 (DEFAULT_TABLE table=test-table [id=34b76f764cc44165ac3f028e5bce850a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:23.568602 2546 tablet_service.cc:1467] Processing CreateTablet for tablet ebe1b842a5354c94b5c486f72e108e28 (DEFAULT_TABLE table=test-table [id=34b76f764cc44165ac3f028e5bce850a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:23.568602 2407 tablet_service.cc:1467] Processing CreateTablet for tablet ebe1b842a5354c94b5c486f72e108e28 (DEFAULT_TABLE table=test-table [id=34b76f764cc44165ac3f028e5bce850a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:23.569767 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe1b842a5354c94b5c486f72e108e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:23.569868 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe1b842a5354c94b5c486f72e108e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:23.569864 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebe1b842a5354c94b5c486f72e108e28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:23.580899 2635 tablet_bootstrap.cc:492] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: Bootstrap starting.
I20260620 15:16:23.584020 2637 tablet_bootstrap.cc:492] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: Bootstrap starting.
I20260620 15:16:23.584599 2635 tablet_bootstrap.cc:654] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:23.586015 2635 log.cc:824] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:23.587774 2637 tablet_bootstrap.cc:654] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:23.589159 2636 tablet_bootstrap.cc:492] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: Bootstrap starting.
I20260620 15:16:23.589306 2637 log.cc:824] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:23.591913 2635 tablet_bootstrap.cc:492] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: No bootstrap required, opened a new log
I20260620 15:16:23.592181 2635 ts_tablet_manager.cc:1397] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260620 15:16:23.592731 2636 tablet_bootstrap.cc:654] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:23.593422 2637 tablet_bootstrap.cc:492] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: No bootstrap required, opened a new log
I20260620 15:16:23.593675 2637 ts_tablet_manager.cc:1397] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 15:16:23.593904 2636 log.cc:824] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:23.596701 2636 tablet_bootstrap.cc:492] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: No bootstrap required, opened a new log
I20260620 15:16:23.596939 2636 ts_tablet_manager.cc:1397] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:16:23.600584 2635 raft_consensus.cc:348] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } }
I20260620 15:16:23.600930 2635 raft_consensus.cc:374] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:23.601048 2635 raft_consensus.cc:729] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f2fec3be6c47ea821688657a1cac4a, State: Initialized, Role: FOLLOWER
I20260620 15:16:23.601629 2635 consensus_queue.cc:260] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } }
I20260620 15:16:23.603842 2637 raft_consensus.cc:348] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } }
I20260620 15:16:23.604220 2637 raft_consensus.cc:374] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:23.604300 2637 raft_consensus.cc:729] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb720c0dd61f4e69ab048c03575c3b73, State: Initialized, Role: FOLLOWER
I20260620 15:16:23.604337 2635 ts_tablet_manager.cc:1428] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260620 15:16:23.604916 2637 consensus_queue.cc:260] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } }
I20260620 15:16:23.605957 2616 heartbeater.cc:503] Master 127.0.97.126:43667 was elected leader, sending a full tablet report...
I20260620 15:16:23.606459 2637 ts_tablet_manager.cc:1428] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260620 15:16:23.607052 2636 raft_consensus.cc:348] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } }
I20260620 15:16:23.607445 2636 raft_consensus.cc:374] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:23.607563 2636 raft_consensus.cc:729] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d912d63708b14563b77d0f1990dc681e, State: Initialized, Role: FOLLOWER
I20260620 15:16:23.608177 2636 consensus_queue.cc:260] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } }
I20260620 15:16:23.609699 2477 heartbeater.cc:503] Master 127.0.97.126:43667 was elected leader, sending a full tablet report...
I20260620 15:16:23.610329 2636 ts_tablet_manager.cc:1428] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260620 15:16:23.718668 2617 tablet.cc:2369] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:16:23.720620 2478 tablet.cc:2369] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:16:23.725232 2339 tablet.cc:2369] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:16:23.739691 2643 raft_consensus.cc:482] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:23.740011 2643 raft_consensus.cc:504] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } }
I20260620 15:16:23.741806 2643 leader_election.cc:288] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709), 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711)
I20260620 15:16:23.748309 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.742116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:38197 (local address 127.0.97.67:40709)
0620 15:16:23.742367 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.742377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.743025 (+ 648us) server_negotiation.cc:407] Connection header received
0620 15:16:23.743245 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.743249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.743319 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.743413 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:23.743975 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.744733 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.746572 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.747423 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.747560 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.747667 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.747937 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.748056 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:16:23.748668 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.741964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40709 (local address 127.0.97.66:38197)
0620 15:16:23.742917 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.742933 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.743099 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.743427 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.743432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.743463 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:23.743834 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.743842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.744851 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.744859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.746433 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.746444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.747619 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.747625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.747783 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.747805 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.747915 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.748041 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":22,"thread_start_us":876,"threads_started":1}
I20260620 15:16:23.748672 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.742367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:55361 (local address 127.0.97.65:42711)
0620 15:16:23.742585 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.742594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.743234 (+ 640us) server_negotiation.cc:407] Connection header received
0620 15:16:23.743437 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.743441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.743506 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.743601 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:23.744158 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.744907 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.746805 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.747719 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.747880 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.747990 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.748293 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.748412 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:16:23.748814 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.742104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42711 (local address 127.0.97.66:55361)
0620 15:16:23.743136 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.743149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.743304 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.743614 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.743618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.743645 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:23.744022 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.744030 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.745040 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.745045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.746669 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.746680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.747940 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.747946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.748141 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.748167 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.748269 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.748389 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":85,"thread_start_us":466,"threads_started":1}
I20260620 15:16:23.749651 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe1b842a5354c94b5c486f72e108e28" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb720c0dd61f4e69ab048c03575c3b73" is_pre_election: true
I20260620 15:16:23.749822 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe1b842a5354c94b5c486f72e108e28" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f2fec3be6c47ea821688657a1cac4a" is_pre_election: true
I20260620 15:16:23.750141 2567 raft_consensus.cc:2454] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 0.
I20260620 15:16:23.750335 2293 raft_consensus.cc:2454] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 0.
I20260620 15:16:23.750774 2359 leader_election.cc:302] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d912d63708b14563b77d0f1990dc681e, eb720c0dd61f4e69ab048c03575c3b73; no voters:
I20260620 15:16:23.751147 2643 raft_consensus.cc:2781] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:23.751305 2643 raft_consensus.cc:482] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:23.751430 2643 raft_consensus.cc:3037] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:23.753683 2643 raft_consensus.cc:504] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } }
I20260620 15:16:23.754221 2643 leader_election.cc:288] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 election: Requested vote from peers eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709), 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711)
I20260620 15:16:23.754678 2567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe1b842a5354c94b5c486f72e108e28" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb720c0dd61f4e69ab048c03575c3b73"
I20260620 15:16:23.754678 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebe1b842a5354c94b5c486f72e108e28" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f2fec3be6c47ea821688657a1cac4a"
I20260620 15:16:23.754887 2293 raft_consensus.cc:3037] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:23.754889 2567 raft_consensus.cc:3037] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:23.758025 2567 raft_consensus.cc:2454] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 1.
I20260620 15:16:23.758019 2293 raft_consensus.cc:2454] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 1.
I20260620 15:16:23.758369 2359 leader_election.cc:302] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d912d63708b14563b77d0f1990dc681e, eb720c0dd61f4e69ab048c03575c3b73; no voters:
I20260620 15:16:23.758684 2643 raft_consensus.cc:2781] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:23.759135 2643 raft_consensus.cc:686] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 1 LEADER]: Becoming Leader. State: Replica: d912d63708b14563b77d0f1990dc681e, State: Running, Role: LEADER
I20260620 15:16:23.759625 2643 consensus_queue.cc:237] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } }
I20260620 15:16:23.764359 2132 catalog_manager.cc:5495] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e reported cstate change: term changed from 0 to 1, leader changed from <none> to d912d63708b14563b77d0f1990dc681e (127.0.97.66). New cstate: current_term: 1 leader_uuid: "d912d63708b14563b77d0f1990dc681e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } health_report { overall_health: HEALTHY } } }
Jun 20 15:16:23 dist-test-slave-7s64 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:23.819633 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.806807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.0.97.126:43667)
0620 15:16:23.806964 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.806968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.807491 (+ 523us) server_negotiation.cc:407] Connection header received
0620 15:16:23.807680 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.807685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.807743 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.807838 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:23.808956 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.809733 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.810981 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.811382 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.811544 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.814605 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:23.814609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:23.814612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:23.814636 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:23.817094 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:23.817722 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:23.817729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:23.817734 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:23.817846 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:23.818183 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:23.818191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:23.818195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:23.818440 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:23.818580 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.818947 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.819255 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":52}
I20260620 15:16:23.819659 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.806901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43667 (local address 127.0.0.1:45710)
0620 15:16:23.807390 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.807406 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.807561 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.807860 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.807864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.808581 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:23.808822 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.808830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.809857 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.809865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.810846 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.810855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.811557 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.811563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.811741 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.812329 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:23.812345 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:23.814418 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:23.817218 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:23.817230 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:23.817237 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:23.817607 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:23.817959 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:23.817963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:23.817967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:23.818087 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:23.818563 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:23.818570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:23.818722 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.818878 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.819139 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":1,"thread_start_us":268,"threads_started":1}
I20260620 15:16:23.824344 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:45710:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:16:23.825253 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:23.831450 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7055393382487e8b5047be90b7ffed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82853661ef21427f8ce20a6a5ab6add2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:16:23.831779 2407 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7055393382487e8b5047be90b7ffed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82853661ef21427f8ce20a6a5ab6add2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:16:23.832305 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7055393382487e8b5047be90b7ffed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:23.832593 2407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7055393382487e8b5047be90b7ffed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:23.837018 2546 tablet_service.cc:1467] Processing CreateTablet for tablet 9b7055393382487e8b5047be90b7ffed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82853661ef21427f8ce20a6a5ab6add2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:16:23.837872 2546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b7055393382487e8b5047be90b7ffed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:23.838366 2635 tablet_bootstrap.cc:492] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: Bootstrap starting.
I20260620 15:16:23.838496 2636 tablet_bootstrap.cc:492] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: Bootstrap starting.
I20260620 15:16:23.839986 2635 tablet_bootstrap.cc:654] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:23.840021 2636 tablet_bootstrap.cc:654] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:23.842612 2636 tablet_bootstrap.cc:492] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: No bootstrap required, opened a new log
I20260620 15:16:23.842761 2635 tablet_bootstrap.cc:492] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: No bootstrap required, opened a new log
I20260620 15:16:23.842756 2636 ts_tablet_manager.cc:1397] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:16:23.842876 2635 ts_tablet_manager.cc:1397] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 15:16:23.843578 2637 tablet_bootstrap.cc:492] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: Bootstrap starting.
I20260620 15:16:23.843524 2635 raft_consensus.cc:348] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:23.843508 2636 raft_consensus.cc:348] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:23.843724 2635 raft_consensus.cc:374] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:23.843724 2636 raft_consensus.cc:374] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:23.843803 2635 raft_consensus.cc:729] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f2fec3be6c47ea821688657a1cac4a, State: Initialized, Role: FOLLOWER
I20260620 15:16:23.843833 2636 raft_consensus.cc:729] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d912d63708b14563b77d0f1990dc681e, State: Initialized, Role: FOLLOWER
I20260620 15:16:23.844040 2635 consensus_queue.cc:260] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:23.844046 2636 consensus_queue.cc:260] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:23.844465 2643 tablet_replica.cc:439] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } }
I20260620 15:16:23.844753 2643 tablet_replica.cc:442] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:23.844579 2641 tablet_replica.cc:439] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } }
I20260620 15:16:23.844911 2641 tablet_replica.cc:442] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:23.845191 2637 tablet_bootstrap.cc:654] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:23.845224 2636 ts_tablet_manager.cc:1428] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:16:23.846625 2635 ts_tablet_manager.cc:1428] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 15:16:23.847278 2637 tablet_bootstrap.cc:492] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: No bootstrap required, opened a new log
I20260620 15:16:23.847452 2637 ts_tablet_manager.cc:1397] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 15:16:23.848184 2637 raft_consensus.cc:348] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:23.848421 2637 raft_consensus.cc:374] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:23.848563 2637 raft_consensus.cc:729] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb720c0dd61f4e69ab048c03575c3b73, State: Initialized, Role: FOLLOWER
I20260620 15:16:23.848801 2637 consensus_queue.cc:260] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:23.849349 2637 ts_tablet_manager.cc:1428] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 15:16:23.849175 2642 tablet_replica.cc:439] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } }
I20260620 15:16:23.849531 2642 tablet_replica.cc:442] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:23.862162 2642 raft_consensus.cc:482] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:23.862340 2642 raft_consensus.cc:504] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:23.863930 2642 leader_election.cc:288] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711), d912d63708b14563b77d0f1990dc681e (127.0.97.66:33051)
I20260620 15:16:23.869418 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.864393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:41205 (local address 127.0.97.66:33051)
0620 15:16:23.864572 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.864578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.864804 (+ 226us) server_negotiation.cc:407] Connection header received
0620 15:16:23.864981 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.864984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.865036 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.865136 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:23.865506 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.866201 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.867759 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.868560 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.868686 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.868784 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.869051 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.869167 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:16:23.869446 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.864226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33051 (local address 127.0.97.67:41205)
0620 15:16:23.864701 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.864722 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.864862 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.865179 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.865184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.865209 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:23.865379 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.865387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.866327 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.866335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.867601 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.867612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.868706 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.868712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.868903 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.868926 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.869029 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.869152 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":2}
I20260620 15:16:23.870141 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.864305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:60163 (local address 127.0.97.65:42711)
0620 15:16:23.864511 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.864520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.865093 (+ 573us) server_negotiation.cc:407] Connection header received
0620 15:16:23.865272 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.865277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.865335 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.865428 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:23.865926 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.866640 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.868310 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.869125 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.869281 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.869379 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.869807 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.869939 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:16:23.870275 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.864304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42711 (local address 127.0.97.67:60163)
0620 15:16:23.864993 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.865008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.865122 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.865441 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.865445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.865466 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:23.865788 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.865796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.866785 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.866790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.868163 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.868174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.869347 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.869353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.869556 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.869603 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.869829 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.869934 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":646,"threads_started":1}
I20260620 15:16:23.870371 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7055393382487e8b5047be90b7ffed" candidate_uuid: "eb720c0dd61f4e69ab048c03575c3b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d912d63708b14563b77d0f1990dc681e" is_pre_election: true
I20260620 15:16:23.870802 2432 raft_consensus.cc:2454] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb720c0dd61f4e69ab048c03575c3b73 in term 0.
I20260620 15:16:23.870949 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7055393382487e8b5047be90b7ffed" candidate_uuid: "eb720c0dd61f4e69ab048c03575c3b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f2fec3be6c47ea821688657a1cac4a" is_pre_election: true
I20260620 15:16:23.871191 2293 raft_consensus.cc:2454] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb720c0dd61f4e69ab048c03575c3b73 in term 0.
I20260620 15:16:23.871345 2499 leader_election.cc:302] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d912d63708b14563b77d0f1990dc681e, eb720c0dd61f4e69ab048c03575c3b73; no voters:
I20260620 15:16:23.871728 2642 raft_consensus.cc:2781] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:23.871873 2642 raft_consensus.cc:482] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:23.871977 2642 raft_consensus.cc:3037] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:23.874008 2642 raft_consensus.cc:504] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:23.874526 2642 leader_election.cc:288] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [CANDIDATE]: Term 1 election: Requested vote from peers 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711), d912d63708b14563b77d0f1990dc681e (127.0.97.66:33051)
I20260620 15:16:23.874817 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7055393382487e8b5047be90b7ffed" candidate_uuid: "eb720c0dd61f4e69ab048c03575c3b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f2fec3be6c47ea821688657a1cac4a"
I20260620 15:16:23.875046 2293 raft_consensus.cc:3037] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:23.875023 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b7055393382487e8b5047be90b7ffed" candidate_uuid: "eb720c0dd61f4e69ab048c03575c3b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d912d63708b14563b77d0f1990dc681e"
I20260620 15:16:23.875185 2432 raft_consensus.cc:3037] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:23.877924 2293 raft_consensus.cc:2454] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb720c0dd61f4e69ab048c03575c3b73 in term 1.
I20260620 15:16:23.878095 2432 raft_consensus.cc:2454] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb720c0dd61f4e69ab048c03575c3b73 in term 1.
I20260620 15:16:23.878348 2500 leader_election.cc:302] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f2fec3be6c47ea821688657a1cac4a, eb720c0dd61f4e69ab048c03575c3b73; no voters:
I20260620 15:16:23.878674 2642 raft_consensus.cc:2781] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:23.879097 2642 raft_consensus.cc:686] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [term 1 LEADER]: Becoming Leader. State: Replica: eb720c0dd61f4e69ab048c03575c3b73, State: Running, Role: LEADER
I20260620 15:16:23.879622 2642 consensus_queue.cc:237] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:23.883757 2658 tablet_replica.cc:439] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: TxnStatusTablet state changed. Reason: New leader eb720c0dd61f4e69ab048c03575c3b73. Latest consensus state: current_term: 1 leader_uuid: "eb720c0dd61f4e69ab048c03575c3b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } }
I20260620 15:16:23.884053 2658 tablet_replica.cc:442] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:16:23.884604 2660 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:16:23.884873 2660 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:16:23.884287 2131 catalog_manager.cc:5495] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "eb720c0dd61f4e69ab048c03575c3b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } health_report { overall_health: HEALTHY } } }
I20260620 15:16:23.909456 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.903493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40709 (local address 127.0.0.1:55656)
0620 15:16:23.903728 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.903752 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.903875 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.904518 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.904523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.904537 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:16:23.904682 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.904688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.906136 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.906140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.907086 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.907096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.907985 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.907988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.908118 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.908163 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:16:23.909074 (+ 911us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:16:23.909094 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.909156 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.909257 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 15:16:23.909560 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.903889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.0.97.67:40709)
0620 15:16:23.904068 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.904074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.904093 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:16:23.904169 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.904174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.904289 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.904432 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:23.904841 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.906022 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.907212 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.907851 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.907968 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.908335 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:23.908973 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:23.909108 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.909178 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.909262 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:16:23.916095 2293 raft_consensus.cc:1261] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Refusing update from remote peer eb720c0dd61f4e69ab048c03575c3b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:23.916203 2432 raft_consensus.cc:1261] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Refusing update from remote peer eb720c0dd61f4e69ab048c03575c3b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:23.917007 2658 consensus_queue.cc:1035] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:23.917313 2642 consensus_queue.cc:1035] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:23.925096 2643 tablet_replica.cc:439] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: TxnStatusTablet state changed. Reason: New leader eb720c0dd61f4e69ab048c03575c3b73. Latest consensus state: current_term: 1 leader_uuid: "eb720c0dd61f4e69ab048c03575c3b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } }
I20260620 15:16:23.925424 2643 tablet_replica.cc:442] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:23.927917 2663 mvcc.cc:205] Tried to move back new op lower bound from 7298943319712391168 to 7298943319575158784. Current Snapshot: MvccSnapshot[applied={T|T < 7298943319712391168}]
I20260620 15:16:23.931305 2643 tablet_replica.cc:439] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb720c0dd61f4e69ab048c03575c3b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } }
I20260620 15:16:23.931659 2643 tablet_replica.cc:442] T 9b7055393382487e8b5047be90b7ffed P d912d63708b14563b77d0f1990dc681e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:23.937007 2658 tablet_replica.cc:439] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb720c0dd61f4e69ab048c03575c3b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } }
I20260620 15:16:23.937297 2658 tablet_replica.cc:442] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:16:23.941260 2641 tablet_replica.cc:439] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: TxnStatusTablet state changed. Reason: New leader eb720c0dd61f4e69ab048c03575c3b73. Latest consensus state: current_term: 1 leader_uuid: "eb720c0dd61f4e69ab048c03575c3b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } }
I20260620 15:16:23.941568 2641 tablet_replica.cc:442] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:23.943665 2642 tablet_replica.cc:439] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb720c0dd61f4e69ab048c03575c3b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } }
I20260620 15:16:23.943996 2642 tablet_replica.cc:442] T 9b7055393382487e8b5047be90b7ffed P eb720c0dd61f4e69ab048c03575c3b73: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:16:23.949059 2641 tablet_replica.cc:439] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb720c0dd61f4e69ab048c03575c3b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } } }
I20260620 15:16:23.949285 2641 tablet_replica.cc:442] T 9b7055393382487e8b5047be90b7ffed P 12f2fec3be6c47ea821688657a1cac4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:16:23.953016 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.946132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52358 (local address 127.0.97.66:33051)
0620 15:16:23.946576 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.946585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.946608 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:16:23.946711 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.946730 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.946876 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.947049 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:23.947471 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.948664 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.950470 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.951110 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.951232 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.951840 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:23.952343 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:23.952458 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.952617 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.952682 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260620 15:16:23.969883 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.962931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:40335 (local address 127.0.97.67:40709)
0620 15:16:23.963126 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:23.963132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:23.963688 (+ 556us) server_negotiation.cc:407] Connection header received
0620 15:16:23.963852 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:23.963856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:23.963940 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:23.964018 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:23.964601 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.965366 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.967617 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.968380 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.968562 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.969109 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:23.969259 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:23.969360 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:23.969508 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 15:16:23.969588 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:16:23.970176 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:23.962817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40709 (local address 127.0.97.66:40335)
0620 15:16:23.963580 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:23.963598 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:23.963729 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:23.964057 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:23.964063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:23.964075 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:16:23.964431 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.964442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.965489 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.965494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.967447 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:23.967462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:23.968610 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:23.968616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:23.968812 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:23.968882 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:16:23.969375 (+ 493us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:16:23.969386 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:16:23.969467 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 15:16:23.969607 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":441,"threads_started":1}
I20260620 15:16:23.980813 2293 raft_consensus.cc:1261] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Refusing update from remote peer d912d63708b14563b77d0f1990dc681e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:23.980926 2567 raft_consensus.cc:1261] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Refusing update from remote peer d912d63708b14563b77d0f1990dc681e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:23.981685 2646 consensus_queue.cc:1035] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:23.982090 2643 consensus_queue.cc:1035] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:24.052968 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:24.045966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:36181 (local address 127.0.97.66:33051)
0620 15:16:24.046173 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:24.046179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:24.046848 (+ 669us) server_negotiation.cc:407] Connection header received
0620 15:16:24.047055 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:24.047060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:24.047141 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:24.047216 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:24.048137 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:24.049282 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:24.050839 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:24.051486 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:24.051608 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:24.052187 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:24.052310 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:24.052400 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:24.052588 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 15:16:24.052659 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:16:24.053086 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:24.046003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33051 (local address 127.0.97.67:36181)
0620 15:16:24.046749 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:24.046767 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:24.046920 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:24.047399 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:24.047403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:24.047415 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:16:24.047937 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:24.047958 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:24.049448 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:24.049452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:24.050625 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:24.050637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:24.051671 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:24.051676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:24.051835 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:24.051887 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:16:24.052450 (+ 563us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:16:24.052462 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 15:16:24.052566 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:16:24.052695 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":348,"threads_started":1}
I20260620 15:16:24.093356 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:24.093642 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:24.097291 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320: Sending DeleteTablet for 3 replicas of tablet ebe1b842a5354c94b5c486f72e108e28
I20260620 15:16:24.098241 2546 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe1b842a5354c94b5c486f72e108e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:24 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:55648
I20260620 15:16:24.098532 2407 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe1b842a5354c94b5c486f72e108e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:24 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:52356
I20260620 15:16:24.099359 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet ebe1b842a5354c94b5c486f72e108e28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:24 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40662
I20260620 15:16:24.105079 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:16:24.106797 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:24.116724 2686 tablet_replica.cc:330] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: stopping tablet replica
I20260620 15:16:24.116874 2406 tablet_service.cc:1467] Processing CreateTablet for tablet fa4fe8addabd47358c32261841d3b16c (DEFAULT_TABLE table=test-table [id=185d988aaa5841bf8b0582402032178b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:24.117373 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4fe8addabd47358c32261841d3b16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:24.117298 2545 tablet_service.cc:1467] Processing CreateTablet for tablet fa4fe8addabd47358c32261841d3b16c (DEFAULT_TABLE table=test-table [id=185d988aaa5841bf8b0582402032178b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:24.117748 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4fe8addabd47358c32261841d3b16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:24.122053 2267 tablet_service.cc:1467] Processing CreateTablet for tablet fa4fe8addabd47358c32261841d3b16c (DEFAULT_TABLE table=test-table [id=185d988aaa5841bf8b0582402032178b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:24.122514 2267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa4fe8addabd47358c32261841d3b16c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:24.124817 2637 tablet_bootstrap.cc:492] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73: Bootstrap starting.
I20260620 15:16:24.126513 2637 tablet_bootstrap.cc:654] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:24.129434 2635 tablet_bootstrap.cc:492] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a: Bootstrap starting.
I20260620 15:16:24.129426 2636 tablet_bootstrap.cc:492] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e: Bootstrap starting.
I20260620 15:16:24.131028 2635 tablet_bootstrap.cc:654] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:24.131040 2636 tablet_bootstrap.cc:654] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:24.138801 2686 raft_consensus.cc:2229] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:16:24.139467 2686 raft_consensus.cc:2258] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:24.143626 2686 ts_tablet_manager.cc:1905] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:24.144629 2636 tablet_bootstrap.cc:492] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e: No bootstrap required, opened a new log
I20260620 15:16:24.144907 2636 ts_tablet_manager.cc:1397] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.007s
I20260620 15:16:24.145848 2636 raft_consensus.cc:348] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:24.146260 2635 tablet_bootstrap.cc:492] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a: No bootstrap required, opened a new log
I20260620 15:16:24.146428 2635 ts_tablet_manager.cc:1397] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20260620 15:16:24.147303 2635 raft_consensus.cc:348] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:24.147547 2635 raft_consensus.cc:374] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:24.147632 2635 raft_consensus.cc:729] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f2fec3be6c47ea821688657a1cac4a, State: Initialized, Role: FOLLOWER
I20260620 15:16:24.147473 2636 raft_consensus.cc:374] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:24.148063 2636 raft_consensus.cc:729] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d912d63708b14563b77d0f1990dc681e, State: Initialized, Role: FOLLOWER
I20260620 15:16:24.147924 2635 consensus_queue.cc:260] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:24.148515 2635 ts_tablet_manager.cc:1428] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 15:16:24.149531 2636 consensus_queue.cc:260] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:24.151132 2636 ts_tablet_manager.cc:1428] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 15:16:24.154770 2687 tablet_replica.cc:330] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: stopping tablet replica
I20260620 15:16:24.155260 2687 raft_consensus.cc:2229] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:24.155623 2687 raft_consensus.cc:2258] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:24.157740 2687 ts_tablet_manager.cc:1905] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:24.161224 2637 tablet_bootstrap.cc:492] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73: No bootstrap required, opened a new log
I20260620 15:16:24.161417 2637 ts_tablet_manager.cc:1397] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.008s
I20260620 15:16:24.161796 2682 raft_consensus.cc:482] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:24.161945 2682 raft_consensus.cc:504] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:24.162647 2682 leader_election.cc:288] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711), eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709)
I20260620 15:16:24.163256 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4fe8addabd47358c32261841d3b16c" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f2fec3be6c47ea821688657a1cac4a" is_pre_election: true
I20260620 15:16:24.163472 2293 raft_consensus.cc:2454] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 0.
I20260620 15:16:24.163823 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4fe8addabd47358c32261841d3b16c" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb720c0dd61f4e69ab048c03575c3b73" is_pre_election: true
I20260620 15:16:24.164193 2637 raft_consensus.cc:348] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:24.164453 2637 raft_consensus.cc:374] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:24.164525 2361 leader_election.cc:302] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f2fec3be6c47ea821688657a1cac4a, d912d63708b14563b77d0f1990dc681e; no voters:
I20260620 15:16:24.164574 2637 raft_consensus.cc:729] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb720c0dd61f4e69ab048c03575c3b73, State: Initialized, Role: FOLLOWER
I20260620 15:16:24.164800 2637 consensus_queue.cc:260] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
W20260620 15:16:24.165095 2359 leader_election.cc:341] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709): Illegal state: must be running to vote when last-logged opid is not known
I20260620 15:16:24.165458 2682 raft_consensus.cc:2781] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:24.165572 2682 raft_consensus.cc:482] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:24.165666 2682 raft_consensus.cc:3037] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:24.168358 2637 ts_tablet_manager.cc:1428] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260620 15:16:24.168377 2682 raft_consensus.cc:504] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:24.169270 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4fe8addabd47358c32261841d3b16c" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f2fec3be6c47ea821688657a1cac4a"
I20260620 15:16:24.169487 2293 raft_consensus.cc:3037] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:24.170671 2685 tablet_replica.cc:330] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: stopping tablet replica
I20260620 15:16:24.171073 2685 raft_consensus.cc:2229] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:24.171370 2685 raft_consensus.cc:2258] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:24.172564 2293 raft_consensus.cc:2454] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 1.
I20260620 15:16:24.172992 2361 leader_election.cc:302] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f2fec3be6c47ea821688657a1cac4a, d912d63708b14563b77d0f1990dc681e; no voters:
I20260620 15:16:24.173128 2685 ts_tablet_manager.cc:1905] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:24.173272 2643 raft_consensus.cc:2781] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:24.173391 2643 raft_consensus.cc:686] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 1 LEADER]: Becoming Leader. State: Replica: d912d63708b14563b77d0f1990dc681e, State: Running, Role: LEADER
I20260620 15:16:24.173346 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa4fe8addabd47358c32261841d3b16c" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb720c0dd61f4e69ab048c03575c3b73"
I20260620 15:16:24.173548 2565 raft_consensus.cc:3037] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:24.173619 2643 consensus_queue.cc:237] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:24.176707 2565 raft_consensus.cc:2454] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 1.
I20260620 15:16:24.177217 2682 leader_election.cc:288] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 election: Requested vote from peers 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711), eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709)
I20260620 15:16:24.178791 2687 ts_tablet_manager.cc:1918] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:16:24.178965 2687 log.cc:1196] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/ebe1b842a5354c94b5c486f72e108e28
I20260620 15:16:24.179448 2687 ts_tablet_manager.cc:1939] T ebe1b842a5354c94b5c486f72e108e28 P 12f2fec3be6c47ea821688657a1cac4a: Deleting consensus metadata
I20260620 15:16:24.180720 2686 ts_tablet_manager.cc:1918] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:16:24.180884 2686 log.cc:1196] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/ebe1b842a5354c94b5c486f72e108e28
I20260620 15:16:24.181417 2686 ts_tablet_manager.cc:1939] T ebe1b842a5354c94b5c486f72e108e28 P d912d63708b14563b77d0f1990dc681e: Deleting consensus metadata
I20260620 15:16:24.181476 2119 catalog_manager.cc:4841] TS 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711): tablet ebe1b842a5354c94b5c486f72e108e28 (table test-table [id=34b76f764cc44165ac3f028e5bce850a]) successfully deleted
I20260620 15:16:24.182928 2118 catalog_manager.cc:4841] TS d912d63708b14563b77d0f1990dc681e (127.0.97.66:33051): tablet ebe1b842a5354c94b5c486f72e108e28 (table test-table [id=34b76f764cc44165ac3f028e5bce850a]) successfully deleted
I20260620 15:16:24.186153 2685 ts_tablet_manager.cc:1918] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:16:24.186321 2685 log.cc:1196] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/ebe1b842a5354c94b5c486f72e108e28
I20260620 15:16:24.186192 2130 catalog_manager.cc:5495] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e reported cstate change: term changed from 0 to 1, leader changed from <none> to d912d63708b14563b77d0f1990dc681e (127.0.97.66). New cstate: current_term: 1 leader_uuid: "d912d63708b14563b77d0f1990dc681e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } health_report { overall_health: UNKNOWN } } }
I20260620 15:16:24.186827 2685 ts_tablet_manager.cc:1939] T ebe1b842a5354c94b5c486f72e108e28 P eb720c0dd61f4e69ab048c03575c3b73: Deleting consensus metadata
I20260620 15:16:24.188400 2117 catalog_manager.cc:4841] TS eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709): tablet ebe1b842a5354c94b5c486f72e108e28 (table test-table [id=34b76f764cc44165ac3f028e5bce850a]) successfully deleted
I20260620 15:16:24.212915 2565 raft_consensus.cc:1261] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Refusing update from remote peer d912d63708b14563b77d0f1990dc681e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:24.213171 2293 raft_consensus.cc:1261] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Refusing update from remote peer d912d63708b14563b77d0f1990dc681e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:24.213716 2682 consensus_queue.cc:1035] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:24.214010 2643 consensus_queue.cc:1035] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:24.219169 2664 mvcc.cc:205] Tried to move back new op lower bound from 7298943320931680256 to 7298943320777285632. Current Snapshot: MvccSnapshot[applied={T|T < 7298943320931680256}]
W20260620 15:16:24.240871 2664 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 15:16:25.832839 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:25.833055 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:25.836591 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320: Sending DeleteTablet for 3 replicas of tablet fa4fe8addabd47358c32261841d3b16c
I20260620 15:16:25.837249 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4fe8addabd47358c32261841d3b16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:25 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40662
I20260620 15:16:25.837378 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4fe8addabd47358c32261841d3b16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:25 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:52356
I20260620 15:16:25.837437 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet fa4fe8addabd47358c32261841d3b16c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:25 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:55648
I20260620 15:16:25.848907 2719 tablet_replica.cc:330] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73: stopping tablet replica
I20260620 15:16:25.849162 2719 raft_consensus.cc:2229] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:25.849417 2718 tablet_replica.cc:330] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e: stopping tablet replica
I20260620 15:16:25.849395 2719 raft_consensus.cc:2258] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:25.849593 2718 raft_consensus.cc:2229] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:16:25.849954 2718 raft_consensus.cc:2258] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:25.850484 2719 ts_tablet_manager.cc:1905] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:25.850888 2718 ts_tablet_manager.cc:1905] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:25.857491 2719 ts_tablet_manager.cc:1918] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:16:25.857489 2718 ts_tablet_manager.cc:1918] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:16:25.857607 2718 log.cc:1196] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/fa4fe8addabd47358c32261841d3b16c
I20260620 15:16:25.857729 2719 log.cc:1196] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/fa4fe8addabd47358c32261841d3b16c
I20260620 15:16:25.858047 2718 ts_tablet_manager.cc:1939] T fa4fe8addabd47358c32261841d3b16c P d912d63708b14563b77d0f1990dc681e: Deleting consensus metadata
I20260620 15:16:25.858232 2719 ts_tablet_manager.cc:1939] T fa4fe8addabd47358c32261841d3b16c P eb720c0dd61f4e69ab048c03575c3b73: Deleting consensus metadata
I20260620 15:16:25.859237 2717 tablet_replica.cc:330] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a: stopping tablet replica
I20260620 15:16:25.859234 2117 catalog_manager.cc:4841] TS eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709): tablet fa4fe8addabd47358c32261841d3b16c (table test-table [id=185d988aaa5841bf8b0582402032178b]) successfully deleted
I20260620 15:16:25.859241 2118 catalog_manager.cc:4841] TS d912d63708b14563b77d0f1990dc681e (127.0.97.66:33051): tablet fa4fe8addabd47358c32261841d3b16c (table test-table [id=185d988aaa5841bf8b0582402032178b]) successfully deleted
I20260620 15:16:25.859411 2717 raft_consensus.cc:2229] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:25.859660 2717 raft_consensus.cc:2258] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:25.860452 2717 ts_tablet_manager.cc:1905] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:25.866878 2717 ts_tablet_manager.cc:1918] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:16:25.867022 2717 log.cc:1196] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/fa4fe8addabd47358c32261841d3b16c
I20260620 15:16:25.867405 2717 ts_tablet_manager.cc:1939] T fa4fe8addabd47358c32261841d3b16c P 12f2fec3be6c47ea821688657a1cac4a: Deleting consensus metadata
I20260620 15:16:25.868211 2119 catalog_manager.cc:4841] TS 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711): tablet fa4fe8addabd47358c32261841d3b16c (table test-table [id=185d988aaa5841bf8b0582402032178b]) successfully deleted
Jun 20 15:16:25 dist-test-slave-7s64 krb5kdc[2087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:16:25 dist-test-slave-7s64 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:25.896895 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:25.881348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.0.97.126:43667)
0620 15:16:25.882199 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:25.882204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:25.882221 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:16:25.882267 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:25.882271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:25.882326 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:25.882437 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:25.883704 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:25.884607 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:25.885876 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:25.886387 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:25.886519 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:25.891209 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:25.891216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:25.891221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:25.891280 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:25.894460 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:25.895076 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:25.895083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:25.895087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:25.895218 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:25.895561 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:25.895568 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:25.895571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:25.895840 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:25.896059 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:25.896357 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 15:16:25.896582 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":519,"threads_started":1}
I20260620 15:16:25.900063 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:16:25.900599 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:25.906453 2268 tablet_service.cc:1467] Processing CreateTablet for tablet a1be6a74886940e1adf5e957b841d913 (DEFAULT_TABLE table=test-table [id=76b9b3df882646fe949478d8edd89cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:25.906770 2406 tablet_service.cc:1467] Processing CreateTablet for tablet a1be6a74886940e1adf5e957b841d913 (DEFAULT_TABLE table=test-table [id=76b9b3df882646fe949478d8edd89cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:25.906885 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1be6a74886940e1adf5e957b841d913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:25.906781 2545 tablet_service.cc:1467] Processing CreateTablet for tablet a1be6a74886940e1adf5e957b841d913 (DEFAULT_TABLE table=test-table [id=76b9b3df882646fe949478d8edd89cc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:25.907152 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1be6a74886940e1adf5e957b841d913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:25.907227 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1be6a74886940e1adf5e957b841d913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:25.913247 2728 tablet_bootstrap.cc:492] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73: Bootstrap starting.
I20260620 15:16:25.914288 2729 tablet_bootstrap.cc:492] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a: Bootstrap starting.
I20260620 15:16:25.914680 2730 tablet_bootstrap.cc:492] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e: Bootstrap starting.
I20260620 15:16:25.915350 2728 tablet_bootstrap.cc:654] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:25.915817 2729 tablet_bootstrap.cc:654] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:25.916086 2730 tablet_bootstrap.cc:654] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:25.917735 2729 tablet_bootstrap.cc:492] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a: No bootstrap required, opened a new log
I20260620 15:16:25.917896 2729 ts_tablet_manager.cc:1397] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:16:25.917970 2728 tablet_bootstrap.cc:492] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73: No bootstrap required, opened a new log
I20260620 15:16:25.918104 2728 ts_tablet_manager.cc:1397] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 15:16:25.918756 2730 tablet_bootstrap.cc:492] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e: No bootstrap required, opened a new log
I20260620 15:16:25.918663 2728 raft_consensus.cc:348] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:25.918663 2729 raft_consensus.cc:348] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:25.918859 2728 raft_consensus.cc:374] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:25.918880 2730 ts_tablet_manager.cc:1397] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:16:25.918871 2729 raft_consensus.cc:374] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:25.918942 2728 raft_consensus.cc:729] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb720c0dd61f4e69ab048c03575c3b73, State: Initialized, Role: FOLLOWER
I20260620 15:16:25.918954 2729 raft_consensus.cc:729] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f2fec3be6c47ea821688657a1cac4a, State: Initialized, Role: FOLLOWER
I20260620 15:16:25.919186 2728 consensus_queue.cc:260] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:25.919618 2728 ts_tablet_manager.cc:1428] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 15:16:25.919596 2730 raft_consensus.cc:348] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:25.919183 2729 consensus_queue.cc:260] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:25.919833 2730 raft_consensus.cc:374] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:25.919931 2730 raft_consensus.cc:729] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d912d63708b14563b77d0f1990dc681e, State: Initialized, Role: FOLLOWER
I20260620 15:16:25.920159 2730 consensus_queue.cc:260] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:25.920758 2730 ts_tablet_manager.cc:1428] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:16:25.922477 2729 ts_tablet_manager.cc:1428] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260620 15:16:25.998929 2734 raft_consensus.cc:482] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:25.999158 2734 raft_consensus.cc:504] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.001078 2734 leader_election.cc:288] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d912d63708b14563b77d0f1990dc681e (127.0.97.66:33051), eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709)
I20260620 15:16:26.024533 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:26.001552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:59957 (local address 127.0.97.66:33051)
0620 15:16:26.016745 (+ 15193us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:26.016753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:26.016777 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:16:26.016847 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:26.016852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:26.016923 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:26.017031 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:26.017916 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.019109 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.022232 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.023046 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.023209 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:26.023349 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:26.023677 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 15:16:26.023815 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15046,"thread_start_us":14880,"threads_started":1}
I20260620 15:16:26.024574 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:26.001454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56425 (local address 127.0.97.67:40709)
0620 15:16:26.016925 (+ 15471us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:26.016931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:26.016954 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:16:26.017020 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:26.017024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:26.017088 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:26.017184 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:26.018069 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.019109 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.022204 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.022949 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.023044 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:26.023118 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:26.023617 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 15:16:26.023766 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15355,"thread_start_us":15200,"threads_started":1}
I20260620 15:16:26.024681 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:26.001367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33051 (local address 127.0.97.65:59957)
0620 15:16:26.015829 (+ 14462us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:26.015855 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:26.016012 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:26.017100 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:26.017103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:26.017129 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:26.017671 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:26.017687 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.019260 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.019264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:26.022023 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:26.022035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.023288 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.023296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:26.023447 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:26.023513 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 15:16:26.023624 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 15:16:26.023862 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14286,"mutex_wait_us":338,"thread_start_us":14147,"threads_started":1}
I20260620 15:16:26.024708 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:26.001369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40709 (local address 127.0.97.65:56425)
0620 15:16:26.016120 (+ 14751us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:26.016136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:26.016227 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:26.017248 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:26.017256 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:26.017300 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:26.017889 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:26.017903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.019257 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.019263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:26.022020 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:26.022035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.023156 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.023163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:26.023412 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:26.023488 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 15:16:26.023569 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 15:16:26.023792 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14616,"mutex_wait_us":281,"thread_start_us":14699,"threads_started":1}
I20260620 15:16:26.025276 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1be6a74886940e1adf5e957b841d913" candidate_uuid: "12f2fec3be6c47ea821688657a1cac4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb720c0dd61f4e69ab048c03575c3b73" is_pre_election: true
I20260620 15:16:26.025362 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1be6a74886940e1adf5e957b841d913" candidate_uuid: "12f2fec3be6c47ea821688657a1cac4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d912d63708b14563b77d0f1990dc681e" is_pre_election: true
I20260620 15:16:26.025544 2565 raft_consensus.cc:2454] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12f2fec3be6c47ea821688657a1cac4a in term 0.
I20260620 15:16:26.025592 2432 raft_consensus.cc:2454] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12f2fec3be6c47ea821688657a1cac4a in term 0.
I20260620 15:16:26.026039 2220 leader_election.cc:302] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f2fec3be6c47ea821688657a1cac4a, eb720c0dd61f4e69ab048c03575c3b73; no voters:
I20260620 15:16:26.026415 2734 raft_consensus.cc:2781] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:26.026537 2734 raft_consensus.cc:482] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:26.026629 2734 raft_consensus.cc:3037] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:26.028983 2734 raft_consensus.cc:504] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.029589 2734 leader_election.cc:288] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [CANDIDATE]: Term 1 election: Requested vote from peers d912d63708b14563b77d0f1990dc681e (127.0.97.66:33051), eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709)
I20260620 15:16:26.030004 2432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1be6a74886940e1adf5e957b841d913" candidate_uuid: "12f2fec3be6c47ea821688657a1cac4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d912d63708b14563b77d0f1990dc681e"
I20260620 15:16:26.030206 2432 raft_consensus.cc:3037] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:26.032087 2432 raft_consensus.cc:2454] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12f2fec3be6c47ea821688657a1cac4a in term 1.
I20260620 15:16:26.032536 2221 leader_election.cc:302] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12f2fec3be6c47ea821688657a1cac4a, d912d63708b14563b77d0f1990dc681e; no voters:
I20260620 15:16:26.032500 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1be6a74886940e1adf5e957b841d913" candidate_uuid: "12f2fec3be6c47ea821688657a1cac4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb720c0dd61f4e69ab048c03575c3b73"
I20260620 15:16:26.032750 2565 raft_consensus.cc:3037] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:26.032799 2734 raft_consensus.cc:2781] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:26.033259 2734 raft_consensus.cc:686] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 1 LEADER]: Becoming Leader. State: Replica: 12f2fec3be6c47ea821688657a1cac4a, State: Running, Role: LEADER
I20260620 15:16:26.033661 2734 consensus_queue.cc:237] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.035564 2565 raft_consensus.cc:2454] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12f2fec3be6c47ea821688657a1cac4a in term 1.
I20260620 15:16:26.037170 2131 catalog_manager.cc:5495] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65). New cstate: current_term: 1 leader_uuid: "12f2fec3be6c47ea821688657a1cac4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } health_report { overall_health: UNKNOWN } } }
I20260620 15:16:26.075695 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:26.070019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.0.97.65:42711)
0620 15:16:26.070552 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:26.070557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:26.070576 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:16:26.070655 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:26.070658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:26.070720 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:26.070914 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:26.071326 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.072116 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.073457 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.073880 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.074007 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:26.074452 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:26.075104 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:26.075263 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:26.075312 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 15:16:26.075382 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":282,"threads_started":1}
I20260620 15:16:26.090566 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:26.084949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:44071 (local address 127.0.97.67:40709)
0620 15:16:26.085172 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:26.085179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:26.085509 (+ 330us) server_negotiation.cc:407] Connection header received
0620 15:16:26.085697 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:26.085702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:26.085780 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:26.085872 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:26.086310 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.087027 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.088968 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.089354 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.089489 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:26.089912 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:26.089998 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:26.090090 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:26.090218 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 15:16:26.090295 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:16:26.090658 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:26.084854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40709 (local address 127.0.97.65:44071)
0620 15:16:26.085410 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:26.085429 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:26.085580 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:26.085884 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:26.085889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:26.085899 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:16:26.086165 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:26.086172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.087157 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.087163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:26.088819 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:26.088830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.089515 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.089521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:26.089663 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:26.089708 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:16:26.090090 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:16:26.090103 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 15:16:26.090199 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:16:26.090322 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":285,"threads_started":1}
I20260620 15:16:26.097941 2565 raft_consensus.cc:1261] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Refusing update from remote peer 12f2fec3be6c47ea821688657a1cac4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:26.097926 2432 raft_consensus.cc:1261] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Refusing update from remote peer 12f2fec3be6c47ea821688657a1cac4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:26.098603 2739 consensus_queue.cc:1035] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:26.098980 2734 consensus_queue.cc:1035] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:26.103631 2698 mvcc.cc:205] Tried to move back new op lower bound from 7298943328651948032 to 7298943328398114816. Current Snapshot: MvccSnapshot[applied={T|T < 7298943328651948032}]
I20260620 15:16:26.126403 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:26.120727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:42613 (local address 127.0.97.65:42711)
0620 15:16:26.120953 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:26.120963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:26.121585 (+ 622us) server_negotiation.cc:407] Connection header received
0620 15:16:26.121788 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:26.121797 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:26.121906 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:26.122037 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:26.122425 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.123307 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.124863 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.125422 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.125521 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:26.125877 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:26.125968 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:26.126038 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:26.126135 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 15:16:26.126186 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:16:26.126436 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:26.120695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42711 (local address 127.0.97.67:42613)
0620 15:16:26.121484 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:26.121502 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:26.121630 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:26.122038 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:26.122041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:26.122050 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:16:26.122297 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:26.122304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.123440 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.123448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:26.124742 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:26.124755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.125566 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.125570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:26.125680 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:26.125712 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:16:26.126052 (+ 340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:16:26.126059 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:16:26.126110 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 15:16:26.126193 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":836,"threads_started":1}
I20260620 15:16:26.152421 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:26.152771 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:26.155746 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320: Sending DeleteTablet for 3 replicas of tablet a1be6a74886940e1adf5e957b841d913
I20260620 15:16:26.156610 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet a1be6a74886940e1adf5e957b841d913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:26 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:52356
I20260620 15:16:26.156914 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet a1be6a74886940e1adf5e957b841d913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:26 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40662
I20260620 15:16:26.157121 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet a1be6a74886940e1adf5e957b841d913 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:26 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:55648
I20260620 15:16:26.157186 2718 tablet_replica.cc:330] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e: stopping tablet replica
I20260620 15:16:26.157379 2719 tablet_replica.cc:330] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73: stopping tablet replica
I20260620 15:16:26.157387 2718 raft_consensus.cc:2229] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:26.157563 2718 raft_consensus.cc:2258] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:26.157563 2719 raft_consensus.cc:2229] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:26.157500 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 15:16:26.157735 2719 raft_consensus.cc:2258] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:26.157821 2717 tablet_replica.cc:330] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a: stopping tablet replica
I20260620 15:16:26.158012 2717 raft_consensus.cc:2229] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 1 LEADER]: Raft consensus shutting down.
W20260620 15:16:26.158097 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:26.158447 2717 raft_consensus.cc:2258] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:26.158761 2719 ts_tablet_manager.cc:1905] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:26.160029 2718 ts_tablet_manager.cc:1905] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:26.163822 2719 ts_tablet_manager.cc:1918] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:16:26.163947 2719 log.cc:1196] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/a1be6a74886940e1adf5e957b841d913
I20260620 15:16:26.164327 2719 ts_tablet_manager.cc:1939] T a1be6a74886940e1adf5e957b841d913 P eb720c0dd61f4e69ab048c03575c3b73: Deleting consensus metadata
I20260620 15:16:26.165333 2117 catalog_manager.cc:4841] TS eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709): tablet a1be6a74886940e1adf5e957b841d913 (table test-table [id=76b9b3df882646fe949478d8edd89cc1]) successfully deleted
I20260620 15:16:26.166363 2718 ts_tablet_manager.cc:1918] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:16:26.166492 2718 log.cc:1196] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/a1be6a74886940e1adf5e957b841d913
I20260620 15:16:26.166922 2718 ts_tablet_manager.cc:1939] T a1be6a74886940e1adf5e957b841d913 P d912d63708b14563b77d0f1990dc681e: Deleting consensus metadata
I20260620 15:16:26.167876 2118 catalog_manager.cc:4841] TS d912d63708b14563b77d0f1990dc681e (127.0.97.66:33051): tablet a1be6a74886940e1adf5e957b841d913 (table test-table [id=76b9b3df882646fe949478d8edd89cc1]) successfully deleted
I20260620 15:16:26.168462 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 73482d6506b84662b922334addf20442 (DEFAULT_TABLE table=test-table [id=32608479e6fc486db5c71e0c29d23026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:26.168713 2406 tablet_service.cc:1467] Processing CreateTablet for tablet 73482d6506b84662b922334addf20442 (DEFAULT_TABLE table=test-table [id=32608479e6fc486db5c71e0c29d23026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:26.168920 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73482d6506b84662b922334addf20442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:26.168962 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 73482d6506b84662b922334addf20442 (DEFAULT_TABLE table=test-table [id=32608479e6fc486db5c71e0c29d23026]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:26.169098 2406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73482d6506b84662b922334addf20442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:26.169291 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73482d6506b84662b922334addf20442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:26.173627 2717 ts_tablet_manager.cc:1905] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:26.176345 2729 tablet_bootstrap.cc:492] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a: Bootstrap starting.
I20260620 15:16:26.177088 2728 tablet_bootstrap.cc:492] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73: Bootstrap starting.
I20260620 15:16:26.178011 2729 tablet_bootstrap.cc:654] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:26.179037 2728 tablet_bootstrap.cc:654] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:26.184108 2717 ts_tablet_manager.cc:1918] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:16:26.188684 2728 tablet_bootstrap.cc:492] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73: No bootstrap required, opened a new log
I20260620 15:16:26.189042 2717 log.cc:1196] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/a1be6a74886940e1adf5e957b841d913
I20260620 15:16:26.189060 2730 tablet_bootstrap.cc:492] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e: Bootstrap starting.
I20260620 15:16:26.188328 2729 tablet_bootstrap.cc:492] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a: No bootstrap required, opened a new log
I20260620 15:16:26.189471 2729 ts_tablet_manager.cc:1397] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260620 15:16:26.189893 2717 ts_tablet_manager.cc:1939] T a1be6a74886940e1adf5e957b841d913 P 12f2fec3be6c47ea821688657a1cac4a: Deleting consensus metadata
I20260620 15:16:26.190246 2729 raft_consensus.cc:348] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.190500 2729 raft_consensus.cc:374] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:26.190583 2729 raft_consensus.cc:729] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f2fec3be6c47ea821688657a1cac4a, State: Initialized, Role: FOLLOWER
I20260620 15:16:26.190760 2730 tablet_bootstrap.cc:654] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:26.190832 2729 consensus_queue.cc:260] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.191679 2729 ts_tablet_manager.cc:1428] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 15:16:26.192394 2119 catalog_manager.cc:4841] TS 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711): tablet a1be6a74886940e1adf5e957b841d913 (table test-table [id=76b9b3df882646fe949478d8edd89cc1]) successfully deleted
I20260620 15:16:26.192188 2728 ts_tablet_manager.cc:1397] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260620 15:16:26.194028 2730 tablet_bootstrap.cc:492] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e: No bootstrap required, opened a new log
I20260620 15:16:26.194202 2730 ts_tablet_manager.cc:1397] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 15:16:26.194922 2730 raft_consensus.cc:348] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.195147 2730 raft_consensus.cc:374] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:26.195250 2730 raft_consensus.cc:729] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d912d63708b14563b77d0f1990dc681e, State: Initialized, Role: FOLLOWER
I20260620 15:16:26.195523 2730 consensus_queue.cc:260] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.196035 2730 ts_tablet_manager.cc:1428] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:16:26.196725 2728 raft_consensus.cc:348] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.196977 2728 raft_consensus.cc:374] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:26.197060 2728 raft_consensus.cc:729] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb720c0dd61f4e69ab048c03575c3b73, State: Initialized, Role: FOLLOWER
I20260620 15:16:26.197288 2728 consensus_queue.cc:260] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.197815 2728 ts_tablet_manager.cc:1428] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260620 15:16:26.299544 2710 raft_consensus.cc:482] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:26.299801 2710 raft_consensus.cc:504] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.300614 2710 leader_election.cc:288] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711), eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709)
I20260620 15:16:26.301183 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73482d6506b84662b922334addf20442" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb720c0dd61f4e69ab048c03575c3b73" is_pre_election: true
I20260620 15:16:26.301157 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73482d6506b84662b922334addf20442" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f2fec3be6c47ea821688657a1cac4a" is_pre_election: true
I20260620 15:16:26.301473 2565 raft_consensus.cc:2454] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 0.
I20260620 15:16:26.301504 2292 raft_consensus.cc:2454] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 0.
I20260620 15:16:26.301913 2359 leader_election.cc:302] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d912d63708b14563b77d0f1990dc681e, eb720c0dd61f4e69ab048c03575c3b73; no voters:
I20260620 15:16:26.302219 2710 raft_consensus.cc:2781] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:26.302348 2710 raft_consensus.cc:482] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:26.302470 2710 raft_consensus.cc:3037] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:26.304723 2710 raft_consensus.cc:504] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.305476 2710 leader_election.cc:288] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 election: Requested vote from peers 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711), eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709)
I20260620 15:16:26.305737 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73482d6506b84662b922334addf20442" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12f2fec3be6c47ea821688657a1cac4a"
I20260620 15:16:26.305946 2292 raft_consensus.cc:3037] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:26.306075 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73482d6506b84662b922334addf20442" candidate_uuid: "d912d63708b14563b77d0f1990dc681e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb720c0dd61f4e69ab048c03575c3b73"
I20260620 15:16:26.306314 2565 raft_consensus.cc:3037] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:26.308302 2565 raft_consensus.cc:2454] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 1.
I20260620 15:16:26.308712 2359 leader_election.cc:302] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d912d63708b14563b77d0f1990dc681e, eb720c0dd61f4e69ab048c03575c3b73; no voters:
I20260620 15:16:26.308983 2710 raft_consensus.cc:2781] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:26.309155 2710 raft_consensus.cc:686] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 1 LEADER]: Becoming Leader. State: Replica: d912d63708b14563b77d0f1990dc681e, State: Running, Role: LEADER
I20260620 15:16:26.309437 2710 consensus_queue.cc:237] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } }
I20260620 15:16:26.309675 2292 raft_consensus.cc:2454] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d912d63708b14563b77d0f1990dc681e in term 1.
I20260620 15:16:26.312136 2131 catalog_manager.cc:5495] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e reported cstate change: term changed from 0 to 1, leader changed from <none> to d912d63708b14563b77d0f1990dc681e (127.0.97.66). New cstate: current_term: 1 leader_uuid: "d912d63708b14563b77d0f1990dc681e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d912d63708b14563b77d0f1990dc681e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 } health_report { overall_health: UNKNOWN } } }
I20260620 15:16:26.351588 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:26.347063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.0.97.66:33051)
0620 15:16:26.347246 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:26.347255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:26.347272 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:16:26.347339 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:26.347344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:26.347412 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:26.347527 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:26.347917 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.348666 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.349952 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:26.350358 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:26.350487 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:26.350887 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:26.350998 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:26.351092 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:26.351233 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 15:16:26.351306 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:16:26.359803 2565 raft_consensus.cc:1261] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Refusing update from remote peer d912d63708b14563b77d0f1990dc681e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:26.360641 2292 raft_consensus.cc:1261] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Refusing update from remote peer d912d63708b14563b77d0f1990dc681e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:26.360944 2710 consensus_queue.cc:1035] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb720c0dd61f4e69ab048c03575c3b73" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:26.361166 2682 consensus_queue.cc:1035] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f2fec3be6c47ea821688657a1cac4a" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:16:26.383113 2699 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 15:16:27.955175 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:27.955435 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:27.959342 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8df8c35c3be8406782dd524de95e6320: Sending DeleteTablet for 3 replicas of tablet 73482d6506b84662b922334addf20442
I20260620 15:16:27.959980 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet 73482d6506b84662b922334addf20442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:27 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40662
I20260620 15:16:27.960121 2406 tablet_service.cc:1514] Processing DeleteTablet for tablet 73482d6506b84662b922334addf20442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:27 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:52356
I20260620 15:16:27.960173 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 73482d6506b84662b922334addf20442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:27 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:55648
I20260620 15:16:27.960819 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 2203
I20260620 15:16:27.963512 2782 tablet_replica.cc:330] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e: stopping tablet replica
I20260620 15:16:27.963768 2782 raft_consensus.cc:2229] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:16:27.964195 2782 raft_consensus.cc:2258] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:27.964841 2783 tablet_replica.cc:330] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73: stopping tablet replica
I20260620 15:16:27.964960 2781 tablet_replica.cc:330] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a: stopping tablet replica
I20260620 15:16:27.965085 2783 raft_consensus.cc:2229] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:27.965159 2781 raft_consensus.cc:2229] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:27.965325 2783 raft_consensus.cc:2258] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:27.965397 2781 raft_consensus.cc:2258] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:27.965675 2782 ts_tablet_manager.cc:1905] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:27.966274 2783 ts_tablet_manager.cc:1905] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:27.966274 2781 ts_tablet_manager.cc:1905] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:27.973059 2783 ts_tablet_manager.cc:1918] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:16:27.973232 2783 log.cc:1196] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/73482d6506b84662b922334addf20442
I20260620 15:16:27.973714 2783 ts_tablet_manager.cc:1939] T 73482d6506b84662b922334addf20442 P eb720c0dd61f4e69ab048c03575c3b73: Deleting consensus metadata
I20260620 15:16:27.974670 2117 catalog_manager.cc:4841] TS eb720c0dd61f4e69ab048c03575c3b73 (127.0.97.67:40709): tablet 73482d6506b84662b922334addf20442 (table test-table [id=32608479e6fc486db5c71e0c29d23026]) successfully deleted
I20260620 15:16:27.975463 2782 ts_tablet_manager.cc:1918] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:16:27.975677 2782 log.cc:1196] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/73482d6506b84662b922334addf20442
I20260620 15:16:27.976588 2782 ts_tablet_manager.cc:1939] T 73482d6506b84662b922334addf20442 P d912d63708b14563b77d0f1990dc681e: Deleting consensus metadata
I20260620 15:16:27.977890 2118 catalog_manager.cc:4841] TS d912d63708b14563b77d0f1990dc681e (127.0.97.66:33051): tablet 73482d6506b84662b922334addf20442 (table test-table [id=32608479e6fc486db5c71e0c29d23026]) successfully deleted
I20260620 15:16:27.980679 2781 ts_tablet_manager.cc:1918] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:16:27.980842 2781 log.cc:1196] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/73482d6506b84662b922334addf20442
I20260620 15:16:27.981253 2781 ts_tablet_manager.cc:1939] T 73482d6506b84662b922334addf20442 P 12f2fec3be6c47ea821688657a1cac4a: Deleting consensus metadata
I20260620 15:16:27.982182 2119 catalog_manager.cc:4841] TS 12f2fec3be6c47ea821688657a1cac4a (127.0.97.65:42711): tablet 73482d6506b84662b922334addf20442 (table test-table [id=32608479e6fc486db5c71e0c29d23026]) successfully deleted
Jun 20 15:16:27 dist-test-slave-7s64 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:27.996110 2785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:27.962415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.0.97.65:42711)
0620 15:16:27.981652 (+ 19237us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:27.981656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:27.981673 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:16:27.981712 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:27.981716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:27.981770 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:27.981894 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:27.983276 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:27.983969 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:27.985327 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:27.985743 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:27.985884 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:27.990042 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:27.990060 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:27.990098 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:27.990135 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:27.993930 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:27.994517 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:27.994524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:27.994527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:27.994638 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:27.994967 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:27.994970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:27.994972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:27.995290 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:27.995504 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:27.995709 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 15:16:27.995866 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19137,"thread_start_us":19070,"threads_started":1}
W20260620 15:16:27.996766 2333 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40680
W20260620 15:16:27.997386 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 2203: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:16:27.997587 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 2203
I20260620 15:16:28.019155 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 2342
Jun 20 15:16:28 dist-test-slave-7s64 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:16:28.034354 2786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:28.019849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52378 (local address 127.0.97.66:33051)
0620 15:16:28.020357 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:28.020361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:28.020377 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:16:28.020424 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:28.020427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:28.020528 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:28.020622 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:28.022106 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:28.023095 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:28.024266 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:28.024760 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:28.024859 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:28.028832 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:28.028860 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:28.028874 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:28.028920 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:28.031647 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:28.032276 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:28.032283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:28.032285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:28.032436 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:28.032846 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:28.032851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:28.032854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:28.033220 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:28.033522 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:28.033783 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:16:28.034016 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":265,"threads_started":1}
W20260620 15:16:28.034971 2472 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52378
W20260620 15:16:28.035501 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 2342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:16:28.035627 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 2342
I20260620 15:16:28.057101 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 2481
Jun 20 15:16:28 dist-test-slave-7s64 krb5kdc[2087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968585, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:16:28.071322 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:28.057714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.0.97.67:40709)
0620 15:16:28.058225 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:28.058230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:28.058244 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:16:28.058299 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:28.058302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:28.058358 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:28.058445 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:28.059569 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:28.060266 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:28.061444 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:28.061843 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:28.061932 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:28.065824 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:28.065837 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:28.065853 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:28.065888 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:28.068816 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:28.069389 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:28.069397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:28.069400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:28.069553 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:28.069878 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:28.069881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:28.069883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:28.070236 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:28.070471 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:28.070730 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 15:16:28.070909 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":309,"threads_started":1}
W20260620 15:16:28.071991 2611 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55658
W20260620 15:16:28.072646 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 2481: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:16:28.072791 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 2481
I20260620 15:16:28.094177 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 2100
I20260620 15:16:28.095322 2172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:28.208046 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 2100
2026-06-20T15:16:28Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7983 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:28 dist-test-slave-7s64 krb5kdc[2791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:28 dist-test-slave-7s64 krb5kdc[2791](info): set up 2 sockets
Jun 20 15:16:28 dist-test-slave-7s64 krb5kdc[2791](info): commencing operation
krb5kdc: starting...
W20260620 15:16:29.288691 389 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.010s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:29 dist-test-slave-7s64 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:16:29.460052 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43409
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:29.765525 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:29.765856 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:29.766011 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:29.772964 2804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:29.773092 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:29.773195 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:29.773296 2804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:29.773388 2804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:29.782788 2804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:29.784173 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:29.785457 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:29.797461 2810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:29.797523 2809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:29.797546 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:29.797729 2804 server_base.cc:1034] running on GCE node
I20260620 15:16:29.798794 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:29.800124 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:29.801317 2804 hybrid_clock.cc:648] HybridClock initialized: now 1781968589801263 us; error 40 us; skew 500 ppm
I20260620 15:16:29.801659 2804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:29 dist-test-slave-7s64 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:29.808079 2804 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:16:29.809665 2804 webserver.cc:460] Webserver started at http://127.0.97.126:36879/ using document root <none> and password file <none>
I20260620 15:16:29.810310 2804 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:29.810418 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:29.810688 2804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:29.813710 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "6a3853ef073545329cc9043539279df6"
format_stamp: "Formatted at 2026-06-20 15:16:29 on dist-test-slave-7s64"
server_key: "39795678a0cce0093bf45ebf0f7991eb"
server_key_iv: "4f0a5cc62eefca621ba2fc1a0e37a878"
server_key_version: "encryptionkey@0"
I20260620 15:16:29.814483 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "6a3853ef073545329cc9043539279df6"
format_stamp: "Formatted at 2026-06-20 15:16:29 on dist-test-slave-7s64"
server_key: "39795678a0cce0093bf45ebf0f7991eb"
server_key_iv: "4f0a5cc62eefca621ba2fc1a0e37a878"
server_key_version: "encryptionkey@0"
I20260620 15:16:29.819643 2804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 15:16:29.822913 2819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:29.824188 2804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:16:29.824381 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "6a3853ef073545329cc9043539279df6"
format_stamp: "Formatted at 2026-06-20 15:16:29 on dist-test-slave-7s64"
server_key: "39795678a0cce0093bf45ebf0f7991eb"
server_key_iv: "4f0a5cc62eefca621ba2fc1a0e37a878"
server_key_version: "encryptionkey@0"
I20260620 15:16:29.824646 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:29.844756 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:29.849351 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:29.849659 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:29.869385 2804 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43409
I20260620 15:16:29.869423 2870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43409 every 8 connection(s)
I20260620 15:16:29.870796 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:29.874414 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 2804
I20260620 15:16:29.874629 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:29.874651 2871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:29.875286 389 external_mini_cluster.cc:1428] Setting key 13537c528ae6ca2311de74952553bbc1
I20260620 15:16:29.886422 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6: Bootstrap starting.
Jun 20 15:16:29 dist-test-slave-7s64 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:29.890223 2871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:29.891180 2871 log.cc:824] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:29.893939 2871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6: No bootstrap required, opened a new log
I20260620 15:16:29.898469 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:29.877246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.0.97.126:43409)
0620 15:16:29.877977 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:29.877990 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:29.878028 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:16:29.880474 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:29.880568 (+ 94us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:29.881016 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:29.881441 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:29.882975 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:29.884360 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:29.885758 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:29.886454 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:29.886683 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:29.891208 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:29.891217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:29.891231 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:29.891276 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:29.894463 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:29.895117 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:29.895124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:29.895129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:29.895364 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:29.895766 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:29.895773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:29.895777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:29.896297 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:29.896638 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:29.897073 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 15:16:29.897489 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":312,"threads_started":1}
I20260620 15:16:29.902372 2871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3853ef073545329cc9043539279df6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43409 } }
I20260620 15:16:29.902711 2871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:29.902818 2871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3853ef073545329cc9043539279df6, State: Initialized, Role: FOLLOWER
I20260620 15:16:29.903332 2871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3853ef073545329cc9043539279df6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43409 } }
I20260620 15:16:29.903573 2871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:29.903704 2871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:29.903885 2871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:29.906101 2871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3853ef073545329cc9043539279df6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43409 } }
I20260620 15:16:29.906561 2871 leader_election.cc:302] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a3853ef073545329cc9043539279df6; no voters:
I20260620 15:16:29.907091 2871 leader_election.cc:288] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:29.907229 2876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:29.907877 2876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3853ef073545329cc9043539279df6, State: Running, Role: LEADER
I20260620 15:16:29.908324 2876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3853ef073545329cc9043539279df6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43409 } }
I20260620 15:16:29.909138 2871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:29.911173 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a3853ef073545329cc9043539279df6. Latest consensus state: current_term: 1 leader_uuid: "6a3853ef073545329cc9043539279df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3853ef073545329cc9043539279df6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43409 } } }
I20260620 15:16:29.911576 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:29.911492 2877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a3853ef073545329cc9043539279df6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3853ef073545329cc9043539279df6" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43409 } } }
I20260620 15:16:29.911794 2877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:29.915644 2883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:29.920714 2883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:29.929761 2883 catalog_manager.cc:1347] Generated new cluster ID: d08be66c00e44a3d8be9acd322d71aaa
I20260620 15:16:29.929857 2883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:29.954550 2883 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:29.956251 2883 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:29.969139 2883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a3853ef073545329cc9043539279df6: Generated new TSK 0
I20260620 15:16:29.969969 2883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:16:30.025629 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43409
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:30.348961 2897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:30.349243 2897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:30.349424 2897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:30.356459 2897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:30.356572 2897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:30.356752 2897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:30.365836 2897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:30.367028 2897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:30.368270 2897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:30.380689 2902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:30.380661 2903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:30.380606 2905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:30.381901 2897 server_base.cc:1034] running on GCE node
I20260620 15:16:30.382447 2897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:30.383293 2897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:30.384559 2897 hybrid_clock.cc:648] HybridClock initialized: now 1781968590384479 us; error 63 us; skew 500 ppm
I20260620 15:16:30.384876 2897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:30 dist-test-slave-7s64 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:30.387912 2897 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:16:30.389338 2897 webserver.cc:460] Webserver started at http://127.0.97.65:39357/ using document root <none> and password file <none>
I20260620 15:16:30.390005 2897 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:30.390143 2897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:30.390430 2897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:30.393285 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "0bbfae4b730b44df8d36b402dbcea400"
format_stamp: "Formatted at 2026-06-20 15:16:30 on dist-test-slave-7s64"
server_key: "a145ddbe858634e3b349ea795f3d22e7"
server_key_iv: "d1a24b30fcb427164e1207a9d573806e"
server_key_version: "encryptionkey@0"
I20260620 15:16:30.394027 2897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "0bbfae4b730b44df8d36b402dbcea400"
format_stamp: "Formatted at 2026-06-20 15:16:30 on dist-test-slave-7s64"
server_key: "a145ddbe858634e3b349ea795f3d22e7"
server_key_iv: "d1a24b30fcb427164e1207a9d573806e"
server_key_version: "encryptionkey@0"
I20260620 15:16:30.399120 2897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:16:30.402479 2912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:30.403785 2897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:16:30.403949 2897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "0bbfae4b730b44df8d36b402dbcea400"
format_stamp: "Formatted at 2026-06-20 15:16:30 on dist-test-slave-7s64"
server_key: "a145ddbe858634e3b349ea795f3d22e7"
server_key_iv: "d1a24b30fcb427164e1207a9d573806e"
server_key_version: "encryptionkey@0"
I20260620 15:16:30.404135 2897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:30.422421 2897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:30.426085 2897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:30.426383 2897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:30.427208 2897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:30.428546 2897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:30.428647 2897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:30.428793 2897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:30.428864 2897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:30.462042 2897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:38679
I20260620 15:16:30.462070 3024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:38679 every 8 connection(s)
I20260620 15:16:30.463200 2897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:30.472627 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 2897
I20260620 15:16:30.472820 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:30.473433 389 external_mini_cluster.cc:1428] Setting key 8b6ff794afac1ec99963c053751708cd
Jun 20 15:16:30 dist-test-slave-7s64 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:30.482743 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:30.465524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49575 (local address 127.0.97.126:43409)
0620 15:16:30.465959 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:30.465963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:30.466711 (+ 748us) server_negotiation.cc:407] Connection header received
0620 15:16:30.468548 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:30.468555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:30.468640 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:30.468787 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:30.470174 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:30.470878 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:30.472471 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:30.472995 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:30.473127 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:30.477016 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:30.477021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:30.477024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:30.477053 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:30.479575 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:30.480315 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:30.480323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:30.480328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:30.480472 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:30.480884 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:30.480890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:30.480896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:30.481249 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:30.481398 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:30.482024 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 15:16:30.482265 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":248,"threads_started":1}
I20260620 15:16:30.483242 3028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:30.465914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43409 (local address 127.0.97.65:49575)
0620 15:16:30.466561 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:30.466601 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:30.468250 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:30.469009 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:30.469017 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:30.469509 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:30.470016 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:30.470032 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:30.471026 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:30.471051 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:30.472319 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:30.472331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:30.473206 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:30.473213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:30.473417 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:30.474140 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:30.474173 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:30.476790 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:30.479710 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:30.479721 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:30.479738 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:30.480189 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:30.480602 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:30.480605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:30.480608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:30.480770 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:30.481390 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:30.481395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:30.481667 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 15:16:30.481889 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 15:16:30.482211 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":258,"threads_started":1}
I20260620 15:16:30.484740 3025 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43409
I20260620 15:16:30.485040 3025 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:30.485704 3025 heartbeater.cc:511] Master 127.0.97.126:43409 requested a full tablet report, sending...
I20260620 15:16:30.487792 2836 ts_manager.cc:194] Registered new tserver with Master: 0bbfae4b730b44df8d36b402dbcea400 (127.0.97.65:38679)
I20260620 15:16:30.489223 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:49575
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:16:30.513505 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43409
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:30.819037 3031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:30.819299 3031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:30.819451 3031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:30.826156 3031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:30.826269 3031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:30.826442 3031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:30.835247 3031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:30.836321 3031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:30.837704 3031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:30.849761 3037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:30.849979 3036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:30.850023 3039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:30.851106 3031 server_base.cc:1034] running on GCE node
I20260620 15:16:30.851754 3031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:30.852757 3031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:30.853961 3031 hybrid_clock.cc:648] HybridClock initialized: now 1781968590853906 us; error 53 us; skew 500 ppm
I20260620 15:16:30.854265 3031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:30 dist-test-slave-7s64 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:30.857323 3031 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:16:30.858644 3031 webserver.cc:460] Webserver started at http://127.0.97.66:45093/ using document root <none> and password file <none>
I20260620 15:16:30.859302 3031 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:30.859452 3031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:30.859757 3031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:30.862593 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "3809ce1c28ba4d62aeb56605fc78189f"
format_stamp: "Formatted at 2026-06-20 15:16:30 on dist-test-slave-7s64"
server_key: "f835cb54c97ef3c019b8f48b0c941abd"
server_key_iv: "8e17ee51d055c7312e945970c5e33acf"
server_key_version: "encryptionkey@0"
I20260620 15:16:30.863343 3031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "3809ce1c28ba4d62aeb56605fc78189f"
format_stamp: "Formatted at 2026-06-20 15:16:30 on dist-test-slave-7s64"
server_key: "f835cb54c97ef3c019b8f48b0c941abd"
server_key_iv: "8e17ee51d055c7312e945970c5e33acf"
server_key_version: "encryptionkey@0"
I20260620 15:16:30.868099 3031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 15:16:30.871484 3046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:30.872896 3031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:30.873042 3031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "3809ce1c28ba4d62aeb56605fc78189f"
format_stamp: "Formatted at 2026-06-20 15:16:30 on dist-test-slave-7s64"
server_key: "f835cb54c97ef3c019b8f48b0c941abd"
server_key_iv: "8e17ee51d055c7312e945970c5e33acf"
server_key_version: "encryptionkey@0"
I20260620 15:16:30.873266 3031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:30.890512 3031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:30.894596 3031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:30.894896 3031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:30.895807 3031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:30.897140 3031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:30.897217 3031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:30.897424 3031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:30.897541 3031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 15:16:30.930361 3031 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39523
I20260620 15:16:30.930440 3158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39523 every 8 connection(s)
I20260620 15:16:30.931540 3031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:30.937767 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 3031
I20260620 15:16:30.937963 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:30.938467 389 external_mini_cluster.cc:1428] Setting key d21fe17ee354d9ea3392dea126be3097
Jun 20 15:16:30 dist-test-slave-7s64 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:30.952692 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:30.933823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56367 (local address 127.0.97.126:43409)
0620 15:16:30.933985 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:30.933989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:30.935049 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 15:16:30.936792 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:30.936796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:30.936859 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:30.936949 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:30.938736 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:30.939656 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:30.941767 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:30.942171 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:30.942263 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:30.946093 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:30.946101 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:30.946109 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:30.946152 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:30.949639 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:30.950257 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:30.950265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:30.950271 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:30.950413 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:30.950845 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:30.950853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:30.950859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:30.951194 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:30.951338 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:30.951958 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 15:16:30.952194 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:16:30.953269 3161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:30.934262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43409 (local address 127.0.97.66:56367)
0620 15:16:30.934900 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:30.934939 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:30.936488 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:30.937138 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:30.937145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:30.937805 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:30.938568 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:30.938593 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:30.939801 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:30.939807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:30.941629 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:30.941646 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:30.942396 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:30.942405 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:30.942716 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:30.943388 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:30.943427 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:30.945870 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:30.949760 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:30.949765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:30.949775 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:30.950136 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:30.950557 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:30.950563 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:30.950570 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:30.950718 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:30.951305 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:30.951310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:30.951561 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 15:16:30.951813 (+ 252us) client_negotiation.cc:240] Negotiation successful
0620 15:16:30.952127 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":245,"threads_started":1}
I20260620 15:16:30.954732 3159 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43409
I20260620 15:16:30.955060 3159 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:30.955886 3159 heartbeater.cc:511] Master 127.0.97.126:43409 requested a full tablet report, sending...
I20260620 15:16:30.957496 2836 ts_manager.cc:194] Registered new tserver with Master: 3809ce1c28ba4d62aeb56605fc78189f (127.0.97.66:39523)
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260620 15:16:30.958403 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:56367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:16:30.980089 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43409
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:31.296152 3164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:31.296393 3164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:31.296478 3164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:31.303968 3164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:31.304055 3164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:31.304152 3164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:31.313473 3164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:31.314524 3164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:31.315713 3164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:31.327955 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:31.328032 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:31.328456 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:31.328682 3164 server_base.cc:1034] running on GCE node
I20260620 15:16:31.329391 3164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:31.330152 3164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:31.331336 3164 hybrid_clock.cc:648] HybridClock initialized: now 1781968591331301 us; error 41 us; skew 500 ppm
I20260620 15:16:31.331573 3164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:31 dist-test-slave-7s64 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:31.334445 3164 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:16:31.335623 3164 webserver.cc:460] Webserver started at http://127.0.97.67:40111/ using document root <none> and password file <none>
I20260620 15:16:31.336167 3164 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:31.336253 3164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:31.336476 3164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:31.339044 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "1ee564633fdb4fafa0bc2a1b145e768c"
format_stamp: "Formatted at 2026-06-20 15:16:31 on dist-test-slave-7s64"
server_key: "8ac7e217bd7f53644c02c2d0e95a304c"
server_key_iv: "d9c770dfe826270d99fea194526106a0"
server_key_version: "encryptionkey@0"
I20260620 15:16:31.339736 3164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "1ee564633fdb4fafa0bc2a1b145e768c"
format_stamp: "Formatted at 2026-06-20 15:16:31 on dist-test-slave-7s64"
server_key: "8ac7e217bd7f53644c02c2d0e95a304c"
server_key_iv: "d9c770dfe826270d99fea194526106a0"
server_key_version: "encryptionkey@0"
I20260620 15:16:31.344188 3164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 15:16:31.347352 3179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:31.348655 3164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:31.348851 3164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "1ee564633fdb4fafa0bc2a1b145e768c"
format_stamp: "Formatted at 2026-06-20 15:16:31 on dist-test-slave-7s64"
server_key: "8ac7e217bd7f53644c02c2d0e95a304c"
server_key_iv: "d9c770dfe826270d99fea194526106a0"
server_key_version: "encryptionkey@0"
I20260620 15:16:31.349013 3164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:31.371623 3164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:31.375594 3164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:31.375803 3164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:31.376852 3164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:31.378126 3164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:31.378192 3164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:31.378265 3164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:31.378306 3164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:31.410256 3164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44247
I20260620 15:16:31.410333 3291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44247 every 8 connection(s)
I20260620 15:16:31.411453 3164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:31.421345 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 3164
I20260620 15:16:31.421515 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:31.421947 389 external_mini_cluster.cc:1428] Setting key a0edc83d9755794e6628e8fac3701a66
Jun 20 15:16:31 dist-test-slave-7s64 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:31.429883 3027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:31.413859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:43281 (local address 127.0.97.126:43409)
0620 15:16:31.414039 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:31.414043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:31.415111 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 15:16:31.416985 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:31.416993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:31.417080 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:31.417186 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:31.418597 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:31.419374 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:31.420926 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:31.421327 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:31.421456 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:31.424859 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:31.424865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:31.424869 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:31.424907 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:31.427160 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:31.427738 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:31.427742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:31.427744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:31.427887 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:31.428335 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:31.428339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:31.428341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:31.428658 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:31.428768 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:31.429485 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 15:16:31.429640 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:16:31.430738 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:31.414303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43409 (local address 127.0.97.67:43281)
0620 15:16:31.414935 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:31.414974 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:31.416735 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:31.417405 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:31.417413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:31.417886 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:31.418420 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:31.418435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:31.419524 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:31.419531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:31.420794 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:31.420805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:31.421547 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:31.421553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:31.421770 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:31.422399 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:31.422418 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:31.424704 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:31.427269 (+ 2565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:31.427275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:31.427284 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:31.427641 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:31.428016 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:31.428019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:31.428022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:31.428178 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:31.428760 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:31.428765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:31.429136 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 15:16:31.429361 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 15:16:31.429715 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":241,"threads_started":1}
I20260620 15:16:31.431975 3292 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43409
I20260620 15:16:31.432238 3292 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:31.432921 3292 heartbeater.cc:511] Master 127.0.97.126:43409 requested a full tablet report, sending...
I20260620 15:16:31.434151 2836 ts_manager.cc:194] Registered new tserver with Master: 1ee564633fdb4fafa0bc2a1b145e768c (127.0.97.67:44247)
I20260620 15:16:31.434900 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:43281
I20260620 15:16:31.436980 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:16:31.466483 3027 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:16:31.464006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.0.97.126:43409)
0620 15:16:31.464265 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:31.464273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:31.464638 (+ 365us) server_negotiation.cc:407] Connection header received
0620 15:16:31.464811 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:31.464816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:31.464882 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:31.464975 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:31.466223 (+ 1248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42808: BlockingRecv error: recv got EOF from 127.0.0.1:42808 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:16:31.469672 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 2897
W20260620 15:16:31.472114 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 2897: Not authorized: Client connection negotiation failed: client connection to 127.0.97.65:38679: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:16:31.472339 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 2897
I20260620 15:16:31.488219 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3031
W20260620 15:16:31.490636 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3031: Not authorized: Client connection negotiation failed: client connection to 127.0.97.66:39523: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:16:31.490749 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 3031
I20260620 15:16:31.505692 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3164
W20260620 15:16:31.508353 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3164: Not authorized: Client connection negotiation failed: client connection to 127.0.97.67:44247: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:16:31.508526 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 3164
I20260620 15:16:31.523444 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 2804
I20260620 15:16:31.524452 2866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:31.623333 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 2804
2026-06-20T15:16:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3412 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:31 dist-test-slave-7s64 krb5kdc[3309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:31 dist-test-slave-7s64 krb5kdc[3309](info): set up 2 sockets
Jun 20 15:16:31 dist-test-slave-7s64 krb5kdc[3309](info): commencing operation
krb5kdc: starting...
W20260620 15:16:32.708832 389 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:32 dist-test-slave-7s64 krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:16:32.880038 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39173
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:39173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:33.183811 3322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:33.184124 3322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:33.184285 3322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:33.191299 3322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:33.191430 3322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:33.191532 3322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:33.191637 3322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:33.191725 3322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:33.201149 3322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:39173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:33.202437 3322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:33.203660 3322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:33.214741 3328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:33.214744 3330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:33.215299 3327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:33.215638 3322 server_base.cc:1034] running on GCE node
I20260620 15:16:33.216414 3322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:33.217739 3322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:33.218930 3322 hybrid_clock.cc:648] HybridClock initialized: now 1781968593218880 us; error 51 us; skew 500 ppm
I20260620 15:16:33.219259 3322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:33 dist-test-slave-7s64 krb5kdc[3309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:33.222433 3322 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:16:33.223672 3322 webserver.cc:460] Webserver started at http://127.0.97.126:45349/ using document root <none> and password file <none>
I20260620 15:16:33.224233 3322 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:33.224335 3322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:33.224635 3322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:33.227301 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "bd2dbb70420e4036ab2f2e1488f28908"
format_stamp: "Formatted at 2026-06-20 15:16:33 on dist-test-slave-7s64"
server_key: "1cad46353deea359cd12b805e011e104"
server_key_iv: "cdeedf9510be0129db07e0a20e8bfc92"
server_key_version: "encryptionkey@0"
I20260620 15:16:33.227980 3322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "bd2dbb70420e4036ab2f2e1488f28908"
format_stamp: "Formatted at 2026-06-20 15:16:33 on dist-test-slave-7s64"
server_key: "1cad46353deea359cd12b805e011e104"
server_key_iv: "cdeedf9510be0129db07e0a20e8bfc92"
server_key_version: "encryptionkey@0"
I20260620 15:16:33.232638 3322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260620 15:16:33.235644 3337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:33.237013 3322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:33.237185 3322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "bd2dbb70420e4036ab2f2e1488f28908"
format_stamp: "Formatted at 2026-06-20 15:16:33 on dist-test-slave-7s64"
server_key: "1cad46353deea359cd12b805e011e104"
server_key_iv: "cdeedf9510be0129db07e0a20e8bfc92"
server_key_version: "encryptionkey@0"
I20260620 15:16:33.237377 3322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:33.249500 3322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:33.253505 3322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:33.253777 3322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:33.270938 3322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:39173
I20260620 15:16:33.270980 3388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:39173 every 8 connection(s)
I20260620 15:16:33.272234 3322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:33.275717 3389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:33.276458 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 3322
I20260620 15:16:33.276728 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:33.277515 389 external_mini_cluster.cc:1428] Setting key 36876c1f17c48973e738922fca3bcb2e
I20260620 15:16:33.287393 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908: Bootstrap starting.
Jun 20 15:16:33 dist-test-slave-7s64 krb5kdc[3309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:33.290115 3389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:33.291030 3389 log.cc:824] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:33.293609 3389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908: No bootstrap required, opened a new log
I20260620 15:16:33.297605 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:33.279313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36268 (local address 127.0.97.126:39173)
0620 15:16:33.280038 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:33.280048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:33.280086 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:16:33.281861 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:33.281881 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:33.282233 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:33.282615 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:33.283987 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:33.285355 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:33.286564 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:33.287187 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:33.287367 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:33.291035 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:33.291042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:33.291055 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:33.291090 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:33.294028 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:33.294658 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:33.294664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:33.294670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:33.294826 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:33.295185 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:33.295191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:33.295194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:33.295670 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:33.295984 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:33.296333 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 15:16:33.296722 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":318,"threads_started":1}
I20260620 15:16:33.301036 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:16:33.301167 3389 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2dbb70420e4036ab2f2e1488f28908" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39173 } }
I20260620 15:16:33.301489 3389 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:33.301589 3389 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2dbb70420e4036ab2f2e1488f28908, State: Initialized, Role: FOLLOWER
I20260620 15:16:33.302130 3389 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2dbb70420e4036ab2f2e1488f28908" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39173 } }
I20260620 15:16:33.302362 3389 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:33.302489 3389 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:33.302668 3389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:33.304678 3389 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2dbb70420e4036ab2f2e1488f28908" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39173 } }
I20260620 15:16:33.305116 3389 leader_election.cc:302] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd2dbb70420e4036ab2f2e1488f28908; no voters:
I20260620 15:16:33.305712 3389 leader_election.cc:288] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:33.305800 3394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:33.306885 3394 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [term 1 LEADER]: Becoming Leader. State: Replica: bd2dbb70420e4036ab2f2e1488f28908, State: Running, Role: LEADER
I20260620 15:16:33.307400 3394 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2dbb70420e4036ab2f2e1488f28908" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39173 } }
I20260620 15:16:33.307873 3389 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:33.312671 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd2dbb70420e4036ab2f2e1488f28908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2dbb70420e4036ab2f2e1488f28908" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39173 } } }
I20260620 15:16:33.312698 3396 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd2dbb70420e4036ab2f2e1488f28908. Latest consensus state: current_term: 1 leader_uuid: "bd2dbb70420e4036ab2f2e1488f28908" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2dbb70420e4036ab2f2e1488f28908" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39173 } } }
I20260620 15:16:33.313046 3396 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:33.313046 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:33.317467 3407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:33.322404 3407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:33.330107 3407 catalog_manager.cc:1347] Generated new cluster ID: 238dff7a8576498ab375f28291ab5a53
I20260620 15:16:33.330214 3407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:33.348970 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:33.340128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.0.97.126:39173)
0620 15:16:33.340293 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:33.340299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:33.340954 (+ 655us) server_negotiation.cc:407] Connection header received
0620 15:16:33.341134 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:33.341141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:33.341238 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:33.341314 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:33.342502 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:33.343691 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:33.345425 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:33.346106 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:33.346245 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:33.347369 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:33.347375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:33.347380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:33.348182 (+ 802us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:33.348249 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:33.348276 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:33.348390 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:33.348535 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 15:16:33.348627 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 15:16:33.368744 3407 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:33.370621 3407 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:33.380086 3407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd2dbb70420e4036ab2f2e1488f28908: Generated new TSK 0
I20260620 15:16:33.381131 3407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:16:33.393538 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3322
I20260620 15:16:33.394832 3384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:33.485795 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 3322
2026-06-20T15:16:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1857 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:33 dist-test-slave-7s64 krb5kdc[3424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:33 dist-test-slave-7s64 krb5kdc[3424](info): set up 2 sockets
Jun 20 15:16:33 dist-test-slave-7s64 krb5kdc[3424](info): commencing operation
krb5kdc: starting...
W20260620 15:16:34.560814 389 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:34 dist-test-slave-7s64 krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:16:34.755612 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42673
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:35.062759 3437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:35.063019 3437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:35.063160 3437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:35.070730 3437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:35.070822 3437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:35.070919 3437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:35.071012 3437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:35.071076 3437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:35.080327 3437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:35.081573 3437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:35.082779 3437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:35.093853 3442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:35.093803 3445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:35.093886 3443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:35.094955 3437 server_base.cc:1034] running on GCE node
I20260620 15:16:35.095582 3437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:35.096880 3437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:35.098079 3437 hybrid_clock.cc:648] HybridClock initialized: now 1781968595098038 us; error 27 us; skew 500 ppm
I20260620 15:16:35.098398 3437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:35 dist-test-slave-7s64 krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:35.105528 3437 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:16:35.106984 3437 webserver.cc:460] Webserver started at http://127.0.97.126:33957/ using document root <none> and password file <none>
I20260620 15:16:35.107604 3437 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:35.107733 3437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:35.108029 3437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:35.110888 3437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "34828a1b9fde43e8a6fe64293aae12e0"
format_stamp: "Formatted at 2026-06-20 15:16:35 on dist-test-slave-7s64"
server_key: "8c6aac029cee1473e643ce9088519fdc"
server_key_iv: "747e84695d0e2a2d38fb20a09c84a016"
server_key_version: "encryptionkey@0"
I20260620 15:16:35.111594 3437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "34828a1b9fde43e8a6fe64293aae12e0"
format_stamp: "Formatted at 2026-06-20 15:16:35 on dist-test-slave-7s64"
server_key: "8c6aac029cee1473e643ce9088519fdc"
server_key_iv: "747e84695d0e2a2d38fb20a09c84a016"
server_key_version: "encryptionkey@0"
I20260620 15:16:35.116389 3437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 15:16:35.119592 3452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:35.120983 3437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:35.121153 3437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "34828a1b9fde43e8a6fe64293aae12e0"
format_stamp: "Formatted at 2026-06-20 15:16:35 on dist-test-slave-7s64"
server_key: "8c6aac029cee1473e643ce9088519fdc"
server_key_iv: "747e84695d0e2a2d38fb20a09c84a016"
server_key_version: "encryptionkey@0"
I20260620 15:16:35.121342 3437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:35.142612 3437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:35.146303 3437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:35.146574 3437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:35.163997 3437 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42673
I20260620 15:16:35.164037 3503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42673 every 8 connection(s)
I20260620 15:16:35.165249 3437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:35.168730 3504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:35.171888 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 3437
I20260620 15:16:35.172113 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:35.172948 389 external_mini_cluster.cc:1428] Setting key a6408628b6c43e59cc69e4baa27bb5f6
I20260620 15:16:35.181354 3504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0: Bootstrap starting.
I20260620 15:16:35.184928 3504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:35.186187 3504 log.cc:824] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:16:35 dist-test-slave-7s64 krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:35.189255 3504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0: No bootstrap required, opened a new log
I20260620 15:16:35.195497 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:35.174938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.0.97.126:42673)
0620 15:16:35.175644 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:35.175654 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:35.175693 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:16:35.177397 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:35.177423 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:35.177781 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:35.178090 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:35.179778 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:35.181120 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:35.182804 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:35.183425 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:35.183540 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:35.188638 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:35.188646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:35.188658 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:35.188689 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:35.191704 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:35.192446 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:35.192452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:35.192457 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:35.192659 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:35.193056 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:35.193060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:35.193062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:35.193512 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:35.193735 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:35.194183 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 15:16:35.194504 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":364,"threads_started":1}
I20260620 15:16:35.198050 3504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34828a1b9fde43e8a6fe64293aae12e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42673 } }
I20260620 15:16:35.198487 3504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:35.198566 3504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34828a1b9fde43e8a6fe64293aae12e0, State: Initialized, Role: FOLLOWER
I20260620 15:16:35.199071 3504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34828a1b9fde43e8a6fe64293aae12e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42673 } }
I20260620 15:16:35.199257 3504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:35.199355 3504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:35.199469 3504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:35.201761 3504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34828a1b9fde43e8a6fe64293aae12e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42673 } }
I20260620 15:16:35.202194 3504 leader_election.cc:302] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34828a1b9fde43e8a6fe64293aae12e0; no voters:
I20260620 15:16:35.202685 3504 leader_election.cc:288] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:35.202775 3509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:35.203446 3509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [term 1 LEADER]: Becoming Leader. State: Replica: 34828a1b9fde43e8a6fe64293aae12e0, State: Running, Role: LEADER
I20260620 15:16:35.204278 3509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34828a1b9fde43e8a6fe64293aae12e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42673 } }
I20260620 15:16:35.204833 3504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:35.207226 3510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34828a1b9fde43e8a6fe64293aae12e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34828a1b9fde43e8a6fe64293aae12e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42673 } } }
I20260620 15:16:35.207540 3510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:35.207280 3511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34828a1b9fde43e8a6fe64293aae12e0. Latest consensus state: current_term: 1 leader_uuid: "34828a1b9fde43e8a6fe64293aae12e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34828a1b9fde43e8a6fe64293aae12e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42673 } } }
I20260620 15:16:35.207842 3511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:35.213519 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:35.219501 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:35.227595 3516 catalog_manager.cc:1347] Generated new cluster ID: 591b90b60cf741a3b41085f66cb6e1f5
I20260620 15:16:35.227700 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:35.252882 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:35.254570 3516 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:35.268280 3516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34828a1b9fde43e8a6fe64293aae12e0: Generated new TSK 0
I20260620 15:16:35.269246 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:16:35.315799 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42673
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:35.625761 3530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:35.626001 3530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:35.626140 3530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:35.633006 3530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:35.633090 3530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:35.633272 3530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:35.642347 3530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:35.643448 3530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:35.644640 3530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:35.657025 3535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:35.657109 3536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:35.657138 3538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:35.657917 3530 server_base.cc:1034] running on GCE node
I20260620 15:16:35.658453 3530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:35.659291 3530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:35.660548 3530 hybrid_clock.cc:648] HybridClock initialized: now 1781968595660458 us; error 50 us; skew 500 ppm
I20260620 15:16:35.660851 3530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:35 dist-test-slave-7s64 krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:35.663774 3530 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:16:35.665189 3530 webserver.cc:460] Webserver started at http://127.0.97.65:45469/ using document root <none> and password file <none>
I20260620 15:16:35.665812 3530 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:35.665947 3530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:35.666234 3530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:35.668900 3530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "35aa22ddf02c4c37ae962f59db1fd993"
format_stamp: "Formatted at 2026-06-20 15:16:35 on dist-test-slave-7s64"
server_key: "cb58785260479f91bf71ddfdb56594d6"
server_key_iv: "1ab14121b000f5239d009a92c7d8ac36"
server_key_version: "encryptionkey@0"
I20260620 15:16:35.669631 3530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "35aa22ddf02c4c37ae962f59db1fd993"
format_stamp: "Formatted at 2026-06-20 15:16:35 on dist-test-slave-7s64"
server_key: "cb58785260479f91bf71ddfdb56594d6"
server_key_iv: "1ab14121b000f5239d009a92c7d8ac36"
server_key_version: "encryptionkey@0"
I20260620 15:16:35.674327 3530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:16:35.677532 3545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:35.678781 3530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:35.678987 3530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "35aa22ddf02c4c37ae962f59db1fd993"
format_stamp: "Formatted at 2026-06-20 15:16:35 on dist-test-slave-7s64"
server_key: "cb58785260479f91bf71ddfdb56594d6"
server_key_iv: "1ab14121b000f5239d009a92c7d8ac36"
server_key_version: "encryptionkey@0"
I20260620 15:16:35.679177 3530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:35.698139 3530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:35.701927 3530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:35.702199 3530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:35.703096 3530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:35.704387 3530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:35.704485 3530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:35.704648 3530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:35.704720 3530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:35.737749 3530 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41229
I20260620 15:16:35.737793 3657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41229 every 8 connection(s)
I20260620 15:16:35.738950 3530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:35.741230 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 3530
I20260620 15:16:35.741441 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:35.742190 389 external_mini_cluster.cc:1428] Setting key e17252784a6db5bb955bf7d79f4fbefc
Jun 20 15:16:35 dist-test-slave-7s64 krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:35.757591 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:35.741436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47081 (local address 127.0.97.126:42673)
0620 15:16:35.741907 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:35.741913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:35.744574 (+ 2661us) server_negotiation.cc:407] Connection header received
0620 15:16:35.744678 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:35.744683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:35.744768 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:35.744884 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:35.746267 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:35.746984 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:35.748454 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:35.748909 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:35.749044 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:35.752210 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:35.752215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:35.752218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:35.752246 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:35.754750 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:35.755395 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:35.755402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:35.755408 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:35.755521 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:35.755919 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:35.755926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:35.755929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:35.756208 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:35.756373 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:35.757031 (+ 658us) server_negotiation.cc:299] Negotiation successful
0620 15:16:35.757231 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":229,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:16:35.758492 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:35.742005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42673 (local address 127.0.97.65:47081)
0620 15:16:35.742767 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:35.742826 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:35.744390 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:35.745107 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:35.745116 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:35.745577 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:35.746104 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:35.746120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:35.747124 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:35.747136 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:35.748314 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:35.748325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:35.749126 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:35.749136 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:35.749360 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:35.750008 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:35.750033 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:35.752005 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:35.754883 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:35.754891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:35.754905 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:35.755281 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:35.755624 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:35.755630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:35.755633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:35.755813 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:35.756349 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:35.756356 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:35.756706 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 15:16:35.756960 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 15:16:35.757419 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":325,"threads_started":1}
I20260620 15:16:35.759979 3658 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260620 15:16:35.760257 3658 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:35.761125 3658 heartbeater.cc:511] Master 127.0.97.126:42673 requested a full tablet report, sending...
I20260620 15:16:35.763230 3469 ts_manager.cc:194] Registered new tserver with Master: 35aa22ddf02c4c37ae962f59db1fd993 (127.0.97.65:41229)
I20260620 15:16:35.765089 3469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:47081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:16:35.781114 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42673
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:36.086576 3664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:36.086812 3664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:36.086961 3664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:36.093825 3664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:36.093916 3664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:36.094081 3664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:36.103140 3664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:36.104259 3664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:36.105649 3664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:36.118199 3669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:36.118223 3672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:36.118160 3670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:36.119292 3664 server_base.cc:1034] running on GCE node
I20260620 15:16:36.119876 3664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:36.120813 3664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:36.122035 3664 hybrid_clock.cc:648] HybridClock initialized: now 1781968596121992 us; error 36 us; skew 500 ppm
I20260620 15:16:36.122331 3664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:36 dist-test-slave-7s64 krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:36.125521 3664 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:16:36.126861 3664 webserver.cc:460] Webserver started at http://127.0.97.66:43003/ using document root <none> and password file <none>
I20260620 15:16:36.127483 3664 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:36.127609 3664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:36.127933 3664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:36.130774 3664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "5ec2e6c2ef0d48bba8729dab27849b67"
format_stamp: "Formatted at 2026-06-20 15:16:36 on dist-test-slave-7s64"
server_key: "e1280a90cb1874b8749f5d14bf967894"
server_key_iv: "ea8a24598eb45b671ab7b9143b9172f4"
server_key_version: "encryptionkey@0"
I20260620 15:16:36.131479 3664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "5ec2e6c2ef0d48bba8729dab27849b67"
format_stamp: "Formatted at 2026-06-20 15:16:36 on dist-test-slave-7s64"
server_key: "e1280a90cb1874b8749f5d14bf967894"
server_key_iv: "ea8a24598eb45b671ab7b9143b9172f4"
server_key_version: "encryptionkey@0"
I20260620 15:16:36.136197 3664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 15:16:36.139366 3679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:36.140676 3664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 15:16:36.140867 3664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "5ec2e6c2ef0d48bba8729dab27849b67"
format_stamp: "Formatted at 2026-06-20 15:16:36 on dist-test-slave-7s64"
server_key: "e1280a90cb1874b8749f5d14bf967894"
server_key_iv: "ea8a24598eb45b671ab7b9143b9172f4"
server_key_version: "encryptionkey@0"
I20260620 15:16:36.141031 3664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:36.156096 3664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:36.160023 3664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:36.160346 3664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:36.161326 3664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:36.162647 3664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:36.162737 3664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:36.162925 3664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:36.162981 3664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:36.198426 3664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:43703
I20260620 15:16:36.198462 3791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:43703 every 8 connection(s)
I20260620 15:16:36.199690 3664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:36.206374 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 3664
I20260620 15:16:36.206590 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:36.207036 389 external_mini_cluster.cc:1428] Setting key cb0220bae1325e925eb5773e95bc52be
Jun 20 15:16:36 dist-test-slave-7s64 krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:36.221798 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:36.202095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:55683 (local address 127.0.97.126:42673)
0620 15:16:36.202274 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:36.202278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:36.203368 (+ 1090us) server_negotiation.cc:407] Connection header received
0620 15:16:36.205339 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:36.205345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:36.205441 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:36.205566 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:36.207056 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.207960 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.209809 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.210202 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.210357 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:36.214446 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:36.214454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:36.214462 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:36.214508 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:36.218464 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.219152 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.219161 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.219168 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.219342 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.219776 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.219784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.219791 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.220188 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:36.220370 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:36.221036 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 15:16:36.221309 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:16:36.222453 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:36.202460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42673 (local address 127.0.97.66:55683)
0620 15:16:36.203191 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:36.203236 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:36.205072 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:36.205793 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:36.205802 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:36.206263 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:36.206896 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:36.206913 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.208108 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.208112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:36.209669 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:36.209681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.210447 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.210453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:36.210670 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:36.211414 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:36.211436 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:36.214208 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:36.218594 (+ 4386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:36.218599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:36.218610 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:36.219009 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:36.219460 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:36.219464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:36.219466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:36.219643 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:36.220329 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:36.220335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:36.220662 (+ 327us) client_negotiation.cc:769] Sending connection context
0620 15:16:36.220875 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 15:16:36.221342 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":287,"threads_started":1}
I20260620 15:16:36.223784 3792 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42673
I20260620 15:16:36.224097 3792 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:36.224867 3792 heartbeater.cc:511] Master 127.0.97.126:42673 requested a full tablet report, sending...
I20260620 15:16:36.226269 3469 ts_manager.cc:194] Registered new tserver with Master: 5ec2e6c2ef0d48bba8729dab27849b67 (127.0.97.66:43703)
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260620 15:16:36.227077 3469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:55683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:16:36.249647 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42673
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:36.555646 3797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:36.555893 3797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:36.556044 3797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:36.562963 3797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:36.563053 3797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:36.563217 3797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:36.572278 3797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:36.573455 3797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:36.574682 3797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:36.587812 3803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:36.587852 3805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:36.587894 3802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:36.593825 3797 server_base.cc:1034] running on GCE node
I20260620 15:16:36.594449 3797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:36.595368 3797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:36.596583 3797 hybrid_clock.cc:648] HybridClock initialized: now 1781968596596542 us; error 58 us; skew 500 ppm
I20260620 15:16:36.596887 3797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:36 dist-test-slave-7s64 krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:36.599915 3797 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:16:36.601526 3797 webserver.cc:460] Webserver started at http://127.0.97.67:42447/ using document root <none> and password file <none>
I20260620 15:16:36.602257 3797 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:36.602437 3797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:36.602756 3797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:36.605868 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "47c9664cf1a247d19f51261af56e1d9b"
format_stamp: "Formatted at 2026-06-20 15:16:36 on dist-test-slave-7s64"
server_key: "2487381ad481d7b07c678f352cf3eb64"
server_key_iv: "df670973856d2127ca24ba0a892afa52"
server_key_version: "encryptionkey@0"
I20260620 15:16:36.606670 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "47c9664cf1a247d19f51261af56e1d9b"
format_stamp: "Formatted at 2026-06-20 15:16:36 on dist-test-slave-7s64"
server_key: "2487381ad481d7b07c678f352cf3eb64"
server_key_iv: "df670973856d2127ca24ba0a892afa52"
server_key_version: "encryptionkey@0"
I20260620 15:16:36.611928 3797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 15:16:36.615348 3812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:36.616771 3797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:36.616978 3797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "47c9664cf1a247d19f51261af56e1d9b"
format_stamp: "Formatted at 2026-06-20 15:16:36 on dist-test-slave-7s64"
server_key: "2487381ad481d7b07c678f352cf3eb64"
server_key_iv: "df670973856d2127ca24ba0a892afa52"
server_key_version: "encryptionkey@0"
I20260620 15:16:36.617177 3797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:36.637858 3797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:36.641655 3797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:36.641937 3797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:36.642789 3797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:36.644068 3797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:36.644162 3797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:36.644315 3797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:36.644387 3797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:36.676654 3797 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:33569
I20260620 15:16:36.676676 3924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:33569 every 8 connection(s)
I20260620 15:16:36.677922 3797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:36.684337 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 3797
I20260620 15:16:36.684553 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:36.684965 389 external_mini_cluster.cc:1428] Setting key 0ead1230feabfd9a564da51f06d9c14e
Jun 20 15:16:36 dist-test-slave-7s64 krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:36.697041 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:36.680426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:40369 (local address 127.0.97.126:42673)
0620 15:16:36.680638 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:36.680643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:36.681708 (+ 1065us) server_negotiation.cc:407] Connection header received
0620 15:16:36.683585 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:36.683590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:36.683678 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:36.683788 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:36.685367 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.686314 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.688019 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.688425 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.688582 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:36.691884 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:36.691889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:36.691892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:36.691919 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:36.694275 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.694929 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.694937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.694940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.695053 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.695461 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.695467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.695471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.695716 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:36.695864 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:36.696448 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 15:16:36.696715 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 15:16:36.697870 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:36.680882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42673 (local address 127.0.97.67:40369)
0620 15:16:36.681556 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:36.681596 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:36.683318 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:36.684012 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:36.684021 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:36.684603 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:36.685207 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:36.685225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.686451 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.686458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:36.687884 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:36.687894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.688661 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.688670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:36.688907 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:36.689482 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:36.689506 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:36.691638 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:36.694405 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:36.694412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:36.694425 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:36.694814 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:36.695159 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:36.695173 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:36.695177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:36.695353 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:36.695837 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:36.695844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:36.696131 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 15:16:36.696375 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 15:16:36.696872 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":252,"threads_started":1}
I20260620 15:16:36.699218 3925 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42673
I20260620 15:16:36.699529 3925 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:36.700237 3925 heartbeater.cc:511] Master 127.0.97.126:42673 requested a full tablet report, sending...
I20260620 15:16:36.701604 3469 ts_manager.cc:194] Registered new tserver with Master: 47c9664cf1a247d19f51261af56e1d9b (127.0.97.67:33569)
I20260620 15:16:36.702365 3469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:40369
I20260620 15:16:36.711738 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:16:36 dist-test-slave-7s64 krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 15:16:36 dist-test-slave-7s64 krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:36.757046 3660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:36.743339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.0.97.126:42673)
0620 15:16:36.743518 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:36.743522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:36.743930 (+ 408us) server_negotiation.cc:407] Connection header received
0620 15:16:36.744066 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:36.744070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:36.744127 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:36.744217 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:36.745707 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.746453 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.747650 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.748039 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.748135 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:36.751720 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:36.751724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:36.751727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:36.751759 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:36.754547 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.755224 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.755228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.755231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.755364 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.755708 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.755711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.755713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.756041 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:36.756217 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:36.756465 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 15:16:36.756783 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 15:16:36.757543 3469 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45762
I20260620 15:16:36.759166 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3530
Jun 20 15:16:36 dist-test-slave-7s64 krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:36.767912 3658 heartbeater.cc:503] Master 127.0.97.126:42673 was elected leader, sending a full tablet report...
I20260620 15:16:36.773907 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:36.760148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.0.97.65:41229)
0620 15:16:36.760842 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:36.760850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:36.760876 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:16:36.760933 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:36.760940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:36.761127 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:36.761280 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:36.762302 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.763339 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.764408 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.765063 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.765165 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:36.769004 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:36.769013 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:36.769026 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:36.769090 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:36.771747 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.772344 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.772351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.772353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.772460 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.772810 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.772813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.772815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.773095 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:36.773299 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:36.773466 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 15:16:36.773627 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":406,"threads_started":1}
W20260620 15:16:36.775090 3653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49942
W20260620 15:16:36.776140 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3530: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:16:36.776335 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 3530
I20260620 15:16:36.792126 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3664
Jun 20 15:16:36 dist-test-slave-7s64 krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:16:36.806743 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:36.792974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.0.97.66:43703)
0620 15:16:36.793502 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:36.793509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:36.793532 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:16:36.793582 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:36.793587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:36.793767 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:36.793882 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:36.794933 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.795891 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.797125 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.797641 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.797736 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:36.801700 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:36.801711 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:36.801723 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:36.801751 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:36.804230 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.804988 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.804998 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.805004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.805122 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.805481 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.805484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.805486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.805791 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:36.805997 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:36.806296 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 15:16:36.806470 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":271,"threads_started":1}
W20260620 15:16:36.807974 3787 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42568
W20260620 15:16:36.809108 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3664: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:16:36.809255 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 3664
I20260620 15:16:36.824898 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3797
Jun 20 15:16:36 dist-test-slave-7s64 krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968596, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:16:36.840649 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:36.825795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.0.97.67:33569)
0620 15:16:36.826383 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:36.826390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:36.826420 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:16:36.826476 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:36.826481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:36.826665 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:36.826848 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:36.828146 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.829298 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.830544 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:36.831105 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:36.831249 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:36.835479 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:36.835489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:36.835501 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:36.835530 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:36.838293 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.838843 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.838849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.838851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.838968 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:36.839273 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:36.839276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:36.839278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:36.839610 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:36.839844 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:36.840090 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 15:16:36.840272 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":280,"threads_started":1}
W20260620 15:16:36.841938 3920 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57578
W20260620 15:16:36.843015 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3797: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:16:36.843142 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 3797
I20260620 15:16:36.858373 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3437
I20260620 15:16:36.859302 3499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:36.959558 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 3437
2026-06-20T15:16:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3480 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:37 dist-test-slave-7s64 krb5kdc[3942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:37 dist-test-slave-7s64 krb5kdc[3942](info): set up 2 sockets
Jun 20 15:16:37 dist-test-slave-7s64 krb5kdc[3942](info): commencing operation
krb5kdc: starting...
W20260620 15:16:38.048820 389 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:38 dist-test-slave-7s64 krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:16:38.227373 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44639
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:38.536968 3955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:38.537269 3955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:38.537396 3955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:38.544337 3955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:38.544468 3955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:38.544600 3955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:38.544701 3955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:38.544780 3955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:38.554032 3955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:38.555225 3955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:38.556435 3955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:38.568116 3960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:38.568204 3961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:38.568158 3963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:38.568825 3955 server_base.cc:1034] running on GCE node
I20260620 15:16:38.569581 3955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:38.570931 3955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:38.572149 3955 hybrid_clock.cc:648] HybridClock initialized: now 1781968598572097 us; error 39 us; skew 500 ppm
I20260620 15:16:38.572467 3955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:38 dist-test-slave-7s64 krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:38.575783 3955 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:16:38.577385 3955 webserver.cc:460] Webserver started at http://127.0.97.126:42233/ using document root <none> and password file <none>
I20260620 15:16:38.578076 3955 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:38.578214 3955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:38.578558 3955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:38.581647 3955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "a1d1b78b17b24979a8c273ecabf4368c"
format_stamp: "Formatted at 2026-06-20 15:16:38 on dist-test-slave-7s64"
server_key: "bfbf9e6a0d241899c7186b2a144a538e"
server_key_iv: "0f4e6f9f23e68d2f504128efa1328500"
server_key_version: "encryptionkey@0"
I20260620 15:16:38.582464 3955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "a1d1b78b17b24979a8c273ecabf4368c"
format_stamp: "Formatted at 2026-06-20 15:16:38 on dist-test-slave-7s64"
server_key: "bfbf9e6a0d241899c7186b2a144a538e"
server_key_iv: "0f4e6f9f23e68d2f504128efa1328500"
server_key_version: "encryptionkey@0"
I20260620 15:16:38.587589 3955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:16:38.591027 3970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:38.592370 3955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:16:38.592583 3955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "a1d1b78b17b24979a8c273ecabf4368c"
format_stamp: "Formatted at 2026-06-20 15:16:38 on dist-test-slave-7s64"
server_key: "bfbf9e6a0d241899c7186b2a144a538e"
server_key_iv: "0f4e6f9f23e68d2f504128efa1328500"
server_key_version: "encryptionkey@0"
I20260620 15:16:38.592773 3955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:38.613080 3955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:38.616982 3955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:38.617238 3955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:38.634574 3955 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44639
I20260620 15:16:38.634608 4021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44639 every 8 connection(s)
I20260620 15:16:38.635807 3955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:38.639174 4022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:38.641721 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 3955
I20260620 15:16:38.641918 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:38.642508 389 external_mini_cluster.cc:1428] Setting key 9595b440270e32b3ed3241003e6079a4
I20260620 15:16:38.651196 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c: Bootstrap starting.
I20260620 15:16:38.654395 4022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:38.655292 4022 log.cc:824] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:16:38 dist-test-slave-7s64 krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:38.658046 4022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c: No bootstrap required, opened a new log
I20260620 15:16:38.664906 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:38.644417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.0.97.126:44639)
0620 15:16:38.645147 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:38.645161 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:38.645204 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:16:38.647272 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:38.647298 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:38.647779 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:38.648183 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:38.649522 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:38.651403 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:38.652605 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:38.653580 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:38.653757 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:38.657890 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:38.657898 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:38.657911 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:38.657942 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:38.661055 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:38.661724 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:38.661734 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:38.661740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:38.661893 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:38.662299 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:38.662304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:38.662314 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:38.662783 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:38.663089 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:38.663526 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 15:16:38.663909 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":293,"threads_started":1}
I20260620 15:16:38.665611 4022 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d1b78b17b24979a8c273ecabf4368c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44639 } }
I20260620 15:16:38.665931 4022 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:38.666060 4022 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d1b78b17b24979a8c273ecabf4368c, State: Initialized, Role: FOLLOWER
I20260620 15:16:38.666589 4022 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d1b78b17b24979a8c273ecabf4368c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44639 } }
I20260620 15:16:38.666827 4022 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:38.666972 4022 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:38.667142 4022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:38.669274 4022 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d1b78b17b24979a8c273ecabf4368c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44639 } }
I20260620 15:16:38.669720 4022 leader_election.cc:302] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1d1b78b17b24979a8c273ecabf4368c; no voters:
I20260620 15:16:38.670233 4022 leader_election.cc:288] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:38.670384 4027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:38.672091 4027 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [term 1 LEADER]: Becoming Leader. State: Replica: a1d1b78b17b24979a8c273ecabf4368c, State: Running, Role: LEADER
I20260620 15:16:38.672299 4022 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:38.672623 4027 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d1b78b17b24979a8c273ecabf4368c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44639 } }
I20260620 15:16:38.675354 4029 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1d1b78b17b24979a8c273ecabf4368c. Latest consensus state: current_term: 1 leader_uuid: "a1d1b78b17b24979a8c273ecabf4368c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d1b78b17b24979a8c273ecabf4368c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44639 } } }
I20260620 15:16:38.675760 4029 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:38.676084 4028 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1d1b78b17b24979a8c273ecabf4368c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1d1b78b17b24979a8c273ecabf4368c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44639 } } }
I20260620 15:16:38.676457 4028 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:38.677361 4034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:38.683106 4034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:38.691303 4034 catalog_manager.cc:1347] Generated new cluster ID: f87dbdd1a7204e99938b3ad45fc73bd4
I20260620 15:16:38.691422 4034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:38.702647 4034 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:38.703764 4034 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:38.709924 4034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1d1b78b17b24979a8c273ecabf4368c: Generated new TSK 0
I20260620 15:16:38.710548 4034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:16:38.760057 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44639
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:39.090389 4048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:39.090632 4048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:39.090718 4048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:39.097818 4048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:39.097910 4048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:39.098014 4048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:39.107110 4048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:39.108189 4048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:39.109396 4048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:39.122104 4053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:39.122344 4056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:39.122373 4054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:39.123447 4048 server_base.cc:1034] running on GCE node
I20260620 15:16:39.123955 4048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:39.124792 4048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:39.125982 4048 hybrid_clock.cc:648] HybridClock initialized: now 1781968599125950 us; error 33 us; skew 500 ppm
I20260620 15:16:39.126263 4048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:39 dist-test-slave-7s64 krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:39.129287 4048 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:16:39.130630 4048 webserver.cc:460] Webserver started at http://127.0.97.65:42475/ using document root <none> and password file <none>
I20260620 15:16:39.131228 4048 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:39.131362 4048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:39.131642 4048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:39.135124 4048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "ccecdb4be81a4344b17a3149fcf645fe"
format_stamp: "Formatted at 2026-06-20 15:16:39 on dist-test-slave-7s64"
server_key: "db5d4bc536942bc3a8dd4a6778855251"
server_key_iv: "9bce4f641c46d3464e98ceefe917b45a"
server_key_version: "encryptionkey@0"
I20260620 15:16:39.136199 4048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "ccecdb4be81a4344b17a3149fcf645fe"
format_stamp: "Formatted at 2026-06-20 15:16:39 on dist-test-slave-7s64"
server_key: "db5d4bc536942bc3a8dd4a6778855251"
server_key_iv: "9bce4f641c46d3464e98ceefe917b45a"
server_key_version: "encryptionkey@0"
I20260620 15:16:39.142510 4048 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 15:16:39.145768 4063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:39.147053 4048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:16:39.147243 4048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "ccecdb4be81a4344b17a3149fcf645fe"
format_stamp: "Formatted at 2026-06-20 15:16:39 on dist-test-slave-7s64"
server_key: "db5d4bc536942bc3a8dd4a6778855251"
server_key_iv: "9bce4f641c46d3464e98ceefe917b45a"
server_key_version: "encryptionkey@0"
I20260620 15:16:39.147434 4048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:39.161098 4048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:39.165309 4048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:39.165575 4048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:39.166587 4048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:39.167884 4048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:39.168022 4048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 15:16:39.168162 4048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:39.168273 4048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:39.200959 4048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41663
I20260620 15:16:39.201040 4175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41663 every 8 connection(s)
I20260620 15:16:39.202154 4048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:39.207039 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 4048
I20260620 15:16:39.207226 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:39.207856 389 external_mini_cluster.cc:1428] Setting key f17761ef1cbe01e982f7604d52af787b
Jun 20 15:16:39 dist-test-slave-7s64 krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:39.223901 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:39.204787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:58311 (local address 127.0.97.126:44639)
0620 15:16:39.205226 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:39.205232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:39.206280 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 15:16:39.208465 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:39.208469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:39.208584 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:39.208711 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:39.210597 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:39.211421 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:39.213949 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:39.214360 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:39.214532 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:39.218248 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:39.218253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:39.218257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:39.218286 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:39.221063 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:39.221753 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:39.221758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:39.221764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:39.221885 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:39.222292 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:39.222297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:39.222300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:39.222546 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:39.222695 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:39.223323 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 15:16:39.223539 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":212,"threads_started":1}
I20260620 15:16:39.224838 4179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:39.205346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44639 (local address 127.0.97.65:58311)
0620 15:16:39.206105 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:39.206157 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:39.208158 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:39.208944 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:39.208952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:39.209625 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:39.210406 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:39.210430 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:39.211634 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:39.211641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:39.213795 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:39.213811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:39.214651 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:39.214661 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:39.214874 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:39.215570 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:39.215602 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:39.218041 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:39.221205 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:39.221214 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:39.221235 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:39.221626 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:39.221992 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:39.221998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:39.222002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:39.222174 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:39.222668 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:39.222676 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:39.222973 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 15:16:39.223216 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 15:16:39.223699 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":297,"threads_started":1}
I20260620 15:16:39.226517 4176 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44639
I20260620 15:16:39.226805 4176 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:39.227499 4176 heartbeater.cc:511] Master 127.0.97.126:44639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:16:39.229583 3987 ts_manager.cc:194] Registered new tserver with Master: ccecdb4be81a4344b17a3149fcf645fe (127.0.97.65:41663)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260620 15:16:39.231014 3987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:58311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:16:39.253804 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44639
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:39.569530 4182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:39.569834 4182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:39.570048 4182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:39.577742 4182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:39.577841 4182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:39.578032 4182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:39.587262 4182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:39.588459 4182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:39.589715 4182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:39.602641 4187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:39.602730 4190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:39.602670 4188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:39.603291 4182 server_base.cc:1034] running on GCE node
I20260620 15:16:39.603905 4182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:39.604913 4182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:39.606137 4182 hybrid_clock.cc:648] HybridClock initialized: now 1781968599606078 us; error 48 us; skew 500 ppm
I20260620 15:16:39.606423 4182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:39 dist-test-slave-7s64 krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:39.609287 4182 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:16:39.610556 4182 webserver.cc:460] Webserver started at http://127.0.97.66:38483/ using document root <none> and password file <none>
I20260620 15:16:39.611167 4182 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:39.611259 4182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:39.611490 4182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:39.614137 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "5ae3504899c3453fb39d83d60e3f541a"
format_stamp: "Formatted at 2026-06-20 15:16:39 on dist-test-slave-7s64"
server_key: "269c98a127424a3985034c60f419f3dd"
server_key_iv: "c5501a3baa8817fc85cbe50485f0f649"
server_key_version: "encryptionkey@0"
I20260620 15:16:39.614814 4182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "5ae3504899c3453fb39d83d60e3f541a"
format_stamp: "Formatted at 2026-06-20 15:16:39 on dist-test-slave-7s64"
server_key: "269c98a127424a3985034c60f419f3dd"
server_key_iv: "c5501a3baa8817fc85cbe50485f0f649"
server_key_version: "encryptionkey@0"
I20260620 15:16:39.619541 4182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:16:39.622694 4197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:39.623848 4182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:16:39.624001 4182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "5ae3504899c3453fb39d83d60e3f541a"
format_stamp: "Formatted at 2026-06-20 15:16:39 on dist-test-slave-7s64"
server_key: "269c98a127424a3985034c60f419f3dd"
server_key_iv: "c5501a3baa8817fc85cbe50485f0f649"
server_key_version: "encryptionkey@0"
I20260620 15:16:39.624208 4182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:39.648077 4182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:39.651952 4182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:39.652213 4182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:39.653137 4182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:39.654300 4182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:39.654368 4182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:39.654527 4182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:39.654587 4182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:39.686333 4182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:44389
I20260620 15:16:39.686403 4309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:44389 every 8 connection(s)
I20260620 15:16:39.687472 4182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:39.689935 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 4182
I20260620 15:16:39.690117 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:39.690680 389 external_mini_cluster.cc:1428] Setting key 0cb6b28b0d686013af29664ade33d9f7
Jun 20 15:16:39 dist-test-slave-7s64 krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:39.708462 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:39.689881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51381 (local address 127.0.97.126:44639)
0620 15:16:39.690084 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:39.690093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:39.691487 (+ 1394us) server_negotiation.cc:407] Connection header received
0620 15:16:39.693449 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:39.693457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:39.693552 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:39.693667 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:39.695062 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:39.696220 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:39.698209 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:39.698604 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:39.698699 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:39.702610 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:39.702615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:39.702618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:39.702647 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:39.705634 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:39.706327 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:39.706334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:39.706338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:39.706465 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:39.706874 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:39.706878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:39.706883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:39.707129 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:39.707273 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:39.707928 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 15:16:39.708139 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260620 15:16:39.709410 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:39.690493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44639 (local address 127.0.97.66:51381)
0620 15:16:39.691300 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:39.691348 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:39.693185 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:39.693889 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:39.693898 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:39.694366 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:39.694887 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:39.694903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:39.696401 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:39.696408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:39.698058 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:39.698079 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:39.698809 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:39.698818 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:39.699140 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:39.699813 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:39.699853 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:39.702405 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:39.705790 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:39.705805 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:39.705817 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:39.706199 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:39.706573 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:39.706579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:39.706582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:39.706748 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:39.707249 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:39.707256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:39.707549 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 15:16:39.707818 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 15:16:39.708300 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":341,"threads_started":1}
I20260620 15:16:39.711052 4310 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44639
I20260620 15:16:39.711362 4310 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
I20260620 15:16:39.712186 4310 heartbeater.cc:511] Master 127.0.97.126:44639 requested a full tablet report, sending...
I20260620 15:16:39.713680 3987 ts_manager.cc:194] Registered new tserver with Master: 5ae3504899c3453fb39d83d60e3f541a (127.0.97.66:44389)
I20260620 15:16:39.714432 3987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:51381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:16:39.733470 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44639
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:40.054276 4315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:40.054581 4315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:40.054733 4315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:40.061718 4315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:40.061816 4315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:40.061987 4315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:40.071058 4315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:40.072271 4315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:40.073606 4315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:40.086567 4320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:40.086472 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:40.087150 4323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:40.088232 4315 server_base.cc:1034] running on GCE node
I20260620 15:16:40.088872 4315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:40.089751 4315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:40.090974 4315 hybrid_clock.cc:648] HybridClock initialized: now 1781968600090917 us; error 59 us; skew 500 ppm
I20260620 15:16:40.091272 4315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:40 dist-test-slave-7s64 krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:40.094376 4315 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:16:40.095753 4315 webserver.cc:460] Webserver started at http://127.0.97.67:36849/ using document root <none> and password file <none>
I20260620 15:16:40.096449 4315 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:40.096608 4315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:40.096904 4315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:40.099579 4315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "898e0d66b76a4f2f9119dce268f30e7a"
format_stamp: "Formatted at 2026-06-20 15:16:40 on dist-test-slave-7s64"
server_key: "391774fa350ba7c931a3115b068d5602"
server_key_iv: "8e73bde855c3b5d6fd1e860f01fba1d6"
server_key_version: "encryptionkey@0"
I20260620 15:16:40.100314 4315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "898e0d66b76a4f2f9119dce268f30e7a"
format_stamp: "Formatted at 2026-06-20 15:16:40 on dist-test-slave-7s64"
server_key: "391774fa350ba7c931a3115b068d5602"
server_key_iv: "8e73bde855c3b5d6fd1e860f01fba1d6"
server_key_version: "encryptionkey@0"
I20260620 15:16:40.105353 4315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:16:40.108714 4330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:40.110020 4315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:16:40.110181 4315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "898e0d66b76a4f2f9119dce268f30e7a"
format_stamp: "Formatted at 2026-06-20 15:16:40 on dist-test-slave-7s64"
server_key: "391774fa350ba7c931a3115b068d5602"
server_key_iv: "8e73bde855c3b5d6fd1e860f01fba1d6"
server_key_version: "encryptionkey@0"
I20260620 15:16:40.110363 4315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:40.129288 4315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:40.133322 4315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:40.133576 4315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:40.134516 4315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:40.135708 4315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:40.135810 4315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:40.135959 4315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:40.136035 4315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:40.171653 4315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:43031
I20260620 15:16:40.171692 4442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:43031 every 8 connection(s)
I20260620 15:16:40.172967 4315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:40.179548 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 4315
I20260620 15:16:40.179725 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:40.180255 389 external_mini_cluster.cc:1428] Setting key 133d5ed01f218de31b893b712ca77c28
Jun 20 15:16:40 dist-test-slave-7s64 krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:40.195933 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:40.175959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44639 (local address 127.0.97.67:51211)
0620 15:16:40.176773 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:40.176822 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:40.178541 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:40.179399 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:40.179410 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:40.180121 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:40.180891 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:40.180909 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.182149 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.182159 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:40.183594 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:40.183610 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.184814 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.184819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:40.184999 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:40.185669 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:40.185689 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:40.187946 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:40.192239 (+ 4293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:40.192245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:40.192255 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:40.192693 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:40.193070 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:40.193073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:40.193075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:40.193229 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:40.193996 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:40.194002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:40.194259 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 15:16:40.194449 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 15:16:40.194766 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":355,"threads_started":1}
I20260620 15:16:40.196972 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:40.175479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51211 (local address 127.0.97.126:44639)
0620 15:16:40.175680 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:40.175686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:40.176943 (+ 1257us) server_negotiation.cc:407] Connection header received
0620 15:16:40.178842 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:40.178850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:40.178980 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:40.179114 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:40.181057 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.182004 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.183746 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.184626 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.184777 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:40.188150 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:40.188159 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:40.188165 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:40.188213 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:40.192094 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.192795 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.192803 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.192809 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.192956 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.193339 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.193346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.193352 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.193768 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:40.196028 (+ 2260us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:40.196205 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 15:16:40.196468 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:16:40.198274 4443 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44639
I20260620 15:16:40.198618 4443 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:40.199492 4443 heartbeater.cc:511] Master 127.0.97.126:44639 requested a full tablet report, sending...
I20260620 15:16:40.201234 3987 ts_manager.cc:194] Registered new tserver with Master: 898e0d66b76a4f2f9119dce268f30e7a (127.0.97.67:43031)
I20260620 15:16:40.202577 3987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:51211
I20260620 15:16:40.208077 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:16:40.233963 4176 heartbeater.cc:503] Master 127.0.97.126:44639 was elected leader, sending a full tablet report...
Jun 20 15:16:40 dist-test-slave-7s64 krb5kdc[3942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:16:40 dist-test-slave-7s64 krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:40.267520 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:40.248819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.0.97.65:41663)
0620 15:16:40.249579 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:40.249586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:40.249611 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:16:40.249659 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:40.249665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:40.249856 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:40.249971 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:40.251879 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.252922 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.255086 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.255613 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.255704 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:40.260470 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:40.260513 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:40.260535 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:40.260588 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:40.264590 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.265160 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.265171 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.265177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.265364 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.265720 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.265728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.265734 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.266196 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:40.266526 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:40.266687 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 15:16:40.266969 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":450,"threads_started":1}
Jun 20 15:16:40 dist-test-slave-7s64 krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:40.285784 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:40.272375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.0.97.126:44639)
0620 15:16:40.272583 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:40.272591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:40.272864 (+ 273us) server_negotiation.cc:407] Connection header received
0620 15:16:40.273048 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:40.273053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:40.273155 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:40.273272 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:40.274585 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.275268 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.276457 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.276909 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.277037 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:40.280770 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:40.280775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:40.280778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:40.280806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:40.283295 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.283897 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.283903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.283906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.284030 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.284368 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.284373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.284377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.284757 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:40.284990 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:40.285284 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 15:16:40.285496 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 15:16:40.286402 3987 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60358
I20260620 15:16:40.287681 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 4048
I20260620 15:16:40.300599 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:40.288433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.0.97.65:41663)
0620 15:16:40.288646 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:40.288654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:40.288885 (+ 231us) server_negotiation.cc:407] Connection header received
0620 15:16:40.289005 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:40.289013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:40.289101 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:40.289221 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:40.290235 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.291505 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.292693 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.293543 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.293715 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:40.296111 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:40.296118 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:40.296125 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:40.296168 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:40.298678 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.299216 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.299220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.299222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.299323 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.299626 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.299628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.299630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.299826 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:40.300002 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:40.300197 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 15:16:40.300330 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:16:40.301118 4171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:40.381953 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 4048
I20260620 15:16:40.398515 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 4182
Jun 20 15:16:40 dist-test-slave-7s64 krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:16:40.414888 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:40.399351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.0.97.66:44389)
0620 15:16:40.399905 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:40.399913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:40.399939 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:16:40.399999 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:40.400006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:40.400229 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:40.400438 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:40.401703 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.402702 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.403958 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.404565 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.404703 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:40.408688 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:40.408699 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:40.408713 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:40.408741 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:40.412289 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.412953 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.412960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.412963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.413077 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.413382 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.413385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.413387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.413808 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:40.414066 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:40.414269 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 15:16:40.414449 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":301,"threads_started":1}
I20260620 15:16:40.416111 4305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:40.498554 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 4182
I20260620 15:16:40.514103 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 4315
Jun 20 15:16:40 dist-test-slave-7s64 krb5kdc[3942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:16:40.530589 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:40.515190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.0.97.67:43031)
0620 15:16:40.515823 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:40.515829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:40.515856 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:16:40.515913 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:40.515919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:40.516118 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:40.516242 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:40.517626 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.518610 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.520249 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:40.520999 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:40.521132 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:40.525197 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:40.525207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:40.525220 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:40.525260 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:40.527965 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.528624 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.528631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.528633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.528746 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:40.529095 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:40.529104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:40.529106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:40.529414 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:40.529698 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:40.530018 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 15:16:40.530271 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":331,"threads_started":1}
I20260620 15:16:40.531809 4438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:40.612277 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 4315
I20260620 15:16:40.627781 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 3955
I20260620 15:16:40.628916 4017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:40.728302 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 3955
2026-06-20T15:16:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3768 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:40 dist-test-slave-7s64 krb5kdc[4462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:40 dist-test-slave-7s64 krb5kdc[4462](info): set up 2 sockets
Jun 20 15:16:40 dist-test-slave-7s64 krb5kdc[4462](info): commencing operation
krb5kdc: starting...
W20260620 15:16:41.812949 389 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:41 dist-test-slave-7s64 krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:16:41.988566 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43833
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:42.288161 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:42.288463 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:42.288646 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:42.295550 4475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:42.295675 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:42.295779 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:42.295879 4475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:42.295971 4475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:42.305140 4475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:42.306246 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:42.307441 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:42.319306 4480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:42.319440 4481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:42.319448 4483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:42.320170 4475 server_base.cc:1034] running on GCE node
I20260620 15:16:42.320971 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:42.322292 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:42.323526 4475 hybrid_clock.cc:648] HybridClock initialized: now 1781968602323439 us; error 76 us; skew 500 ppm
I20260620 15:16:42.323897 4475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:42 dist-test-slave-7s64 krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:42.327049 4475 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:16:42.328133 4475 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:42.328276 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:42.328663 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:42.331604 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "07b06859d6db46c68c1662d58ee6dfb3"
format_stamp: "Formatted at 2026-06-20 15:16:42 on dist-test-slave-7s64"
server_key: "b8a6ee3bb90c6ba70d4492b184f11d28"
server_key_iv: "00b4afe32754ebc4597856dfa99ba9bb"
server_key_version: "encryptionkey@0"
I20260620 15:16:42.332435 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "07b06859d6db46c68c1662d58ee6dfb3"
format_stamp: "Formatted at 2026-06-20 15:16:42 on dist-test-slave-7s64"
server_key: "b8a6ee3bb90c6ba70d4492b184f11d28"
server_key_iv: "00b4afe32754ebc4597856dfa99ba9bb"
server_key_version: "encryptionkey@0"
I20260620 15:16:42.337682 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:16:42.341171 4489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:42.342459 4475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:16:42.342660 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "07b06859d6db46c68c1662d58ee6dfb3"
format_stamp: "Formatted at 2026-06-20 15:16:42 on dist-test-slave-7s64"
server_key: "b8a6ee3bb90c6ba70d4492b184f11d28"
server_key_iv: "00b4afe32754ebc4597856dfa99ba9bb"
server_key_version: "encryptionkey@0"
I20260620 15:16:42.342921 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:42.363361 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:42.367244 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:42.367522 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:42.385979 4540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43833 every 8 connection(s)
I20260620 15:16:42.386065 4475 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43833
I20260620 15:16:42.386909 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:42.390313 4541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:42.393460 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 4475
I20260620 15:16:42.393646 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:42.394435 389 external_mini_cluster.cc:1428] Setting key 928cc4119326418d276eb89baedb3702
I20260620 15:16:42.402889 4541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3: Bootstrap starting.
I20260620 15:16:42.405742 4541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:42.406723 4541 log.cc:824] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:16:42 dist-test-slave-7s64 krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:42.409623 4541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3: No bootstrap required, opened a new log
I20260620 15:16:42.417366 4544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:42.396479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.0.97.126:43833)
0620 15:16:42.397322 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:42.397334 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:42.397376 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:16:42.399015 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:42.399036 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:42.399427 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:42.399758 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:42.401565 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:42.402910 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:42.404332 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:42.405069 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:42.405269 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:42.409691 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:42.409704 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:42.409726 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:42.409774 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:42.412984 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:42.413790 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:42.413802 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:42.413807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:42.414023 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:42.414437 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:42.414440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:42.414442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:42.415049 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:42.415316 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:42.415709 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 15:16:42.416205 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":415,"threads_started":1}
I20260620 15:16:42.418213 4541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b06859d6db46c68c1662d58ee6dfb3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43833 } }
I20260620 15:16:42.418551 4541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:42.418671 4541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b06859d6db46c68c1662d58ee6dfb3, State: Initialized, Role: FOLLOWER
I20260620 15:16:42.419279 4541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b06859d6db46c68c1662d58ee6dfb3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43833 } }
I20260620 15:16:42.419554 4541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:42.419688 4541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:42.419844 4541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:42.422077 4541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b06859d6db46c68c1662d58ee6dfb3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43833 } }
I20260620 15:16:42.422472 4541 leader_election.cc:302] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07b06859d6db46c68c1662d58ee6dfb3; no voters:
I20260620 15:16:42.422899 4541 leader_election.cc:288] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:42.423022 4546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:42.423694 4546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [term 1 LEADER]: Becoming Leader. State: Replica: 07b06859d6db46c68c1662d58ee6dfb3, State: Running, Role: LEADER
I20260620 15:16:42.424206 4546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b06859d6db46c68c1662d58ee6dfb3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43833 } }
I20260620 15:16:42.424889 4541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:42.426578 4548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07b06859d6db46c68c1662d58ee6dfb3. Latest consensus state: current_term: 1 leader_uuid: "07b06859d6db46c68c1662d58ee6dfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b06859d6db46c68c1662d58ee6dfb3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43833 } } }
I20260620 15:16:42.426704 4547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07b06859d6db46c68c1662d58ee6dfb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07b06859d6db46c68c1662d58ee6dfb3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43833 } } }
I20260620 15:16:42.427031 4547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:42.427634 4548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:42.427834 4551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:42.432665 4551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:42.442531 4551 catalog_manager.cc:1347] Generated new cluster ID: a803dae119354900acc15380673d2dbf
I20260620 15:16:42.442643 4551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:42.457540 4551 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:42.458670 4551 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:42.465670 4551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3: Generated new TSK 0
I20260620 15:16:42.466342 4551 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:16:42.517601 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43833
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:42.830554 4567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:42.830832 4567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:42.831022 4567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:42.838069 4567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:42.838202 4567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:42.838385 4567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:42.848042 4567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:42.849222 4567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:42.850414 4567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:42.862413 4572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:42.862442 4575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:42.862480 4573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:42.863445 4567 server_base.cc:1034] running on GCE node
I20260620 15:16:42.863984 4567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:42.864830 4567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:42.866016 4567 hybrid_clock.cc:648] HybridClock initialized: now 1781968602865958 us; error 54 us; skew 500 ppm
I20260620 15:16:42.866263 4567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:42 dist-test-slave-7s64 krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:42.869234 4567 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:16:42.870240 4567 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:42.870332 4567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:42.870618 4567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:42.873334 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "183e0fdffa47437cab4382b372540294"
format_stamp: "Formatted at 2026-06-20 15:16:42 on dist-test-slave-7s64"
server_key: "f4fe3781757e5bbdd728cd7eca7d2148"
server_key_iv: "d7a9e8357dd6626671a0ef2e3ed409d1"
server_key_version: "encryptionkey@0"
I20260620 15:16:42.874099 4567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "183e0fdffa47437cab4382b372540294"
format_stamp: "Formatted at 2026-06-20 15:16:42 on dist-test-slave-7s64"
server_key: "f4fe3781757e5bbdd728cd7eca7d2148"
server_key_iv: "d7a9e8357dd6626671a0ef2e3ed409d1"
server_key_version: "encryptionkey@0"
I20260620 15:16:42.878782 4567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:16:42.882181 4581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:42.883436 4567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:16:42.883616 4567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "183e0fdffa47437cab4382b372540294"
format_stamp: "Formatted at 2026-06-20 15:16:42 on dist-test-slave-7s64"
server_key: "f4fe3781757e5bbdd728cd7eca7d2148"
server_key_iv: "d7a9e8357dd6626671a0ef2e3ed409d1"
server_key_version: "encryptionkey@0"
I20260620 15:16:42.883828 4567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:42.903903 4567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:42.907689 4567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:42.907955 4567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:42.908895 4567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:42.910197 4567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:42.910315 4567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:42.910452 4567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:42.910539 4567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:42.943851 4567 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:37571
I20260620 15:16:42.943895 4693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:37571 every 8 connection(s)
I20260620 15:16:42.944729 4567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:42.947060 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 4567
I20260620 15:16:42.947275 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:42.947947 389 external_mini_cluster.cc:1428] Setting key ded41dab5f547197fd02e754e0570b62
Jun 20 15:16:42 dist-test-slave-7s64 krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:42.965377 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:42.946980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:52877 (local address 127.0.97.126:43833)
0620 15:16:42.947464 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:42.947470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:42.948597 (+ 1127us) server_negotiation.cc:407] Connection header received
0620 15:16:42.950961 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:42.950965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:42.951024 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:42.951114 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:42.953141 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:42.953846 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:42.956149 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:42.956602 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:42.956731 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:42.959976 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:42.959980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:42.959984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:42.960012 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:42.962510 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:42.963159 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:42.963165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:42.963169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:42.963275 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:42.963664 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:42.963669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:42.963672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:42.963934 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:42.964079 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:42.964766 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 15:16:42.964988 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":213,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:16:42.966396 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:42.947609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43833 (local address 127.0.97.65:52877)
0620 15:16:42.948405 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:42.948464 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:42.950683 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:42.951384 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:42.951398 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:42.952160 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:42.952968 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:42.952995 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:42.954000 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:42.954008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:42.955985 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:42.955996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:42.956826 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:42.956833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:42.957045 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:42.957702 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:42.957722 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:42.959765 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:42.962641 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:42.962650 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:42.962665 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:42.963047 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:42.963385 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:42.963390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:42.963394 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:42.963553 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:42.964057 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:42.964064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:42.964380 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 15:16:42.964674 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 15:16:42.965180 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":335,"threads_started":1}
I20260620 15:16:42.967927 4694 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260620 15:16:42.968254 4694 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:42.968911 4694 heartbeater.cc:511] Master 127.0.97.126:43833 requested a full tablet report, sending...
I20260620 15:16:42.970798 4506 ts_manager.cc:194] Registered new tserver with Master: 183e0fdffa47437cab4382b372540294 (127.0.97.65:37571)
I20260620 15:16:42.972162 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:52877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:16:42.991982 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43833
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:43.298831 4700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:43.299067 4700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:43.299198 4700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:43.306084 4700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:43.306171 4700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:43.306373 4700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:43.315325 4700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:43.316471 4700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:43.317674 4700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:43.329895 4705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:43.329794 4706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:43.329933 4708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:43.330691 4700 server_base.cc:1034] running on GCE node
I20260620 15:16:43.331221 4700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:43.332067 4700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:43.333273 4700 hybrid_clock.cc:648] HybridClock initialized: now 1781968603333236 us; error 50 us; skew 500 ppm
I20260620 15:16:43.333561 4700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:43 dist-test-slave-7s64 krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:43.336382 4700 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:16:43.337446 4700 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:43.337575 4700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:43.337868 4700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:43.340615 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "d04a5b16903747108483d8522bcae945"
format_stamp: "Formatted at 2026-06-20 15:16:43 on dist-test-slave-7s64"
server_key: "4ffb05a68f4c230eb009d67d39a334f9"
server_key_iv: "ea0a3456fda51cc306dcdd892e78c115"
server_key_version: "encryptionkey@0"
I20260620 15:16:43.341393 4700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "d04a5b16903747108483d8522bcae945"
format_stamp: "Formatted at 2026-06-20 15:16:43 on dist-test-slave-7s64"
server_key: "4ffb05a68f4c230eb009d67d39a334f9"
server_key_iv: "ea0a3456fda51cc306dcdd892e78c115"
server_key_version: "encryptionkey@0"
I20260620 15:16:43.345896 4700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 15:16:43.349153 4714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:43.350358 4700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:16:43.350526 4700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "d04a5b16903747108483d8522bcae945"
format_stamp: "Formatted at 2026-06-20 15:16:43 on dist-test-slave-7s64"
server_key: "4ffb05a68f4c230eb009d67d39a334f9"
server_key_iv: "ea0a3456fda51cc306dcdd892e78c115"
server_key_version: "encryptionkey@0"
I20260620 15:16:43.350720 4700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:43.374135 4700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:43.377705 4700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:43.377966 4700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:43.378839 4700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:43.380064 4700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:43.380177 4700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:43.380300 4700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:43.380404 4700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:43.413486 4700 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:35141
I20260620 15:16:43.413581 4826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:35141 every 8 connection(s)
I20260620 15:16:43.414238 4700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:43.417680 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 4700
I20260620 15:16:43.417850 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:43.418457 389 external_mini_cluster.cc:1428] Setting key 65d12f8ca56609249a23fc5713891ed3
Jun 20 15:16:43 dist-test-slave-7s64 krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:43.434387 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.416538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44633 (local address 127.0.97.126:43833)
0620 15:16:43.416723 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:43.416730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:43.418010 (+ 1280us) server_negotiation.cc:407] Connection header received
0620 15:16:43.419995 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:43.420000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:43.420066 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:43.420148 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:43.421980 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.422660 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.425101 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.425494 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.425648 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.429114 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:43.429119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:43.429122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:43.429150 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:43.431581 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:43.432253 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:43.432265 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:43.432268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:43.432384 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:43.432801 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:43.432806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:43.432809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:43.433062 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:43.433209 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:43.433888 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 15:16:43.434090 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:16:43.435187 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.416995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43833 (local address 127.0.97.66:44633)
0620 15:16:43.417847 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:43.417902 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:43.419747 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:43.420330 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:43.420337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:43.421042 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:43.421786 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:43.421825 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.422808 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.422816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:43.424976 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:43.424992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.425733 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.425740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:43.425969 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.426617 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:43.426637 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:43.428907 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:43.431721 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:43.431729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:43.431742 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:43.432131 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:43.432511 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:43.432517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:43.432521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:43.432687 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:43.433194 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:43.433202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:43.433513 (+ 311us) client_negotiation.cc:769] Sending connection context
0620 15:16:43.433784 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 15:16:43.434226 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":312,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260620 15:16:43.436681 4827 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43833
I20260620 15:16:43.436996 4827 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:43.437670 4827 heartbeater.cc:511] Master 127.0.97.126:43833 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
I20260620 15:16:43.439034 4506 ts_manager.cc:194] Registered new tserver with Master: d04a5b16903747108483d8522bcae945 (127.0.97.66:35141)
I20260620 15:16:43.439832 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:44633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:16:43.459722 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43833
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:43.768693 4832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:43.768927 4832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:43.769016 4832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:43.776177 4832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:43.776279 4832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:43.776384 4832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:43.785356 4832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:43.786406 4832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:43.787631 4832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:43.799852 4837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:43.799800 4840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:43.799880 4838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:43.800788 4832 server_base.cc:1034] running on GCE node
I20260620 15:16:43.801334 4832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:43.802161 4832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:43.803385 4832 hybrid_clock.cc:648] HybridClock initialized: now 1781968603803346 us; error 32 us; skew 500 ppm
I20260620 15:16:43.803679 4832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:43 dist-test-slave-7s64 krb5kdc[4462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:43.806665 4832 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:16:43.807766 4832 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:43.807883 4832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:43.808210 4832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:43.810883 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "37daab7ada2846288060ce10faa63461"
format_stamp: "Formatted at 2026-06-20 15:16:43 on dist-test-slave-7s64"
server_key: "6517dea4c24f0ad66960806d5f7c8b86"
server_key_iv: "71e31cb32d8c03ebc4d414026cc82521"
server_key_version: "encryptionkey@0"
I20260620 15:16:43.811590 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "37daab7ada2846288060ce10faa63461"
format_stamp: "Formatted at 2026-06-20 15:16:43 on dist-test-slave-7s64"
server_key: "6517dea4c24f0ad66960806d5f7c8b86"
server_key_iv: "71e31cb32d8c03ebc4d414026cc82521"
server_key_version: "encryptionkey@0"
I20260620 15:16:43.816085 4832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 15:16:43.819221 4846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:43.820410 4832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:16:43.820616 4832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "37daab7ada2846288060ce10faa63461"
format_stamp: "Formatted at 2026-06-20 15:16:43 on dist-test-slave-7s64"
server_key: "6517dea4c24f0ad66960806d5f7c8b86"
server_key_iv: "71e31cb32d8c03ebc4d414026cc82521"
server_key_version: "encryptionkey@0"
I20260620 15:16:43.820819 4832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:43.849020 4832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:43.853067 4832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:43.853317 4832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:43.854238 4832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:43.855520 4832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:43.855643 4832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:43.855756 4832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:43.855837 4832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:43.887570 4832 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:38191
I20260620 15:16:43.887647 4958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:38191 every 8 connection(s)
I20260620 15:16:43.888338 4832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:43.894802 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 4832
I20260620 15:16:43.894986 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:43.895512 389 external_mini_cluster.cc:1428] Setting key 4f3df48ee86520fc434aaa477556a1ac
Jun 20 15:16:43 dist-test-slave-7s64 krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968603, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:43.906556 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.890539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35523 (local address 127.0.97.126:43833)
0620 15:16:43.890697 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:43.890701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:43.891746 (+ 1045us) server_negotiation.cc:407] Connection header received
0620 15:16:43.893546 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:43.893551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:43.893616 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:43.893705 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:43.895314 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.896102 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.897685 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.898138 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.898267 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.901568 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:43.901572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:43.901576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:43.901605 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:43.904076 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:43.904782 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:43.904785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:43.904788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:43.904894 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:43.905229 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:43.905232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:43.905233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:43.905465 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:43.905571 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:43.906152 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 15:16:43.906300 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:16:43.907289 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.890942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43833 (local address 127.0.97.67:35523)
0620 15:16:43.891574 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:43.891619 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:43.893297 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:43.893888 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:43.893895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:43.894358 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:43.895147 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:43.895172 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.896254 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.896259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:43.897541 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:43.897553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.898362 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.898369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:43.898587 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.899246 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:43.899289 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:43.901381 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:43.904198 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:43.904204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:43.904213 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:43.904678 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:43.904989 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:43.904992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:43.904994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:43.905138 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:43.905564 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:43.905569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:43.905820 (+ 251us) client_negotiation.cc:769] Sending connection context
0620 15:16:43.906024 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 15:16:43.906343 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":257,"threads_started":1}
I20260620 15:16:43.908569 4959 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43833
I20260620 15:16:43.908840 4959 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:43.909458 4959 heartbeater.cc:511] Master 127.0.97.126:43833 requested a full tablet report, sending...
I20260620 15:16:43.910703 4506 ts_manager.cc:194] Registered new tserver with Master: 37daab7ada2846288060ce10faa63461 (127.0.97.67:38191)
I20260620 15:16:43.911470 4506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:35523
I20260620 15:16:43.921815 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:16:43.937811 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.926259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.0.97.126:43833)
0620 15:16:43.926476 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:43.926486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:43.926845 (+ 359us) server_negotiation.cc:407] Connection header received
0620 15:16:43.926941 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:43.926944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:43.927000 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:43.927086 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:43.928327 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.929143 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.930278 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.930734 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.930854 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.932842 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:43.932847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:43.932851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:43.932878 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:43.935466 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:43.936053 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:43.936056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:43.936059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:43.936176 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:43.936475 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:43.936479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:43.936514 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:43.936839 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:43.937016 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:43.937250 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 15:16:43.937443 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:16:43.941259 4506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:16:43.948041 4506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:43.975003 4694 heartbeater.cc:503] Master 127.0.97.126:43833 was elected leader, sending a full tablet report...
I20260620 15:16:43.980082 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.959285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.0.97.67:38191)
0620 15:16:43.971357 (+ 12072us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:43.971364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:43.971913 (+ 549us) server_negotiation.cc:407] Connection header received
0620 15:16:43.972236 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:43.972248 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:43.972515 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:43.972678 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:43.973967 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.975436 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.978015 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.978913 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.979071 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.979297 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:43.979588 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:16:43.979737 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11870,"thread_start_us":11676,"threads_started":1}
I20260620 15:16:43.980150 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.959203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:38191 (local address 127.0.0.1:40260)
0620 15:16:43.971779 (+ 12576us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:43.971814 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:43.972015 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:43.973066 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:43.973077 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:43.973132 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:43.973714 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:43.973733 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.975585 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.975590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:43.977765 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:43.977778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.979132 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.979138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:43.979346 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.979391 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 15:16:43.979543 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 15:16:43.979725 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12318,"mutex_wait_us":44,"spinlock_wait_cycles":3840,"thread_start_us":12310,"threads_started":1}
I20260620 15:16:43.982049 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.972539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.0.97.65:37571)
0620 15:16:43.973728 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:43.973736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:43.974342 (+ 606us) server_negotiation.cc:407] Connection header received
0620 15:16:43.974457 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:43.974462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:43.974594 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:43.974773 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:43.975316 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.977146 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.979823 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.981114 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.981255 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.981473 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:43.981613 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 15:16:43.981756 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":938,"threads_started":1}
I20260620 15:16:43.982069 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.972655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37571 (local address 127.0.0.1:40124)
0620 15:16:43.974186 (+ 1531us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:43.974202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:43.974315 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:43.974756 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:43.974760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:43.974782 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:43.975139 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:43.975152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.977272 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.977280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:43.979647 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:43.979661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.981299 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.981305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:43.981463 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.981497 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:16:43.981581 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 15:16:43.981697 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"mutex_wait_us":25,"thread_start_us":1231,"threads_started":1}
I20260620 15:16:43.983094 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.959216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:35141 (local address 127.0.0.1:43470)
0620 15:16:43.971778 (+ 12562us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:43.971814 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:43.971986 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:43.973071 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:43.973079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:43.973146 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:43.974179 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:43.974192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.977496 (+ 3304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.977502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:43.980677 (+ 3175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:43.980690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.982310 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.982316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:43.982484 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.982514 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:16:43.982604 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 15:16:43.982714 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12442,"mutex_wait_us":466,"spinlock_wait_cycles":13440,"thread_start_us":12117,"threads_started":1}
I20260620 15:16:43.983114 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:43.959478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.0.97.66:35141)
0620 15:16:43.970917 (+ 11439us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:43.970930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:43.972179 (+ 1249us) server_negotiation.cc:407] Connection header received
0620 15:16:43.972287 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:43.972296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:43.972585 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:43.972768 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:43.974341 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.977222 (+ 2881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.980831 (+ 3609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:43.982154 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:43.982278 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:43.982479 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:43.982647 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 15:16:43.982811 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11252,"thread_start_us":11141,"threads_started":1}
I20260620 15:16:43.984292 4894 tablet_service.cc:1467] Processing CreateTablet for tablet 15710d89e46542d89823a9af0ab0b890 (DEFAULT_TABLE table=test-table [id=8d99c4e9cc774162ab9394f6a427ff6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:43.985764 4894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15710d89e46542d89823a9af0ab0b890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:43.985814 4629 tablet_service.cc:1467] Processing CreateTablet for tablet 15710d89e46542d89823a9af0ab0b890 (DEFAULT_TABLE table=test-table [id=8d99c4e9cc774162ab9394f6a427ff6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:43.986133 4762 tablet_service.cc:1467] Processing CreateTablet for tablet 15710d89e46542d89823a9af0ab0b890 (DEFAULT_TABLE table=test-table [id=8d99c4e9cc774162ab9394f6a427ff6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:43.987128 4629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15710d89e46542d89823a9af0ab0b890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:43.987282 4762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15710d89e46542d89823a9af0ab0b890. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:43.998180 4977 tablet_bootstrap.cc:492] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: Bootstrap starting.
I20260620 15:16:44.000957 4979 tablet_bootstrap.cc:492] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: Bootstrap starting.
I20260620 15:16:44.001713 4977 tablet_bootstrap.cc:654] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:44.002511 4978 tablet_bootstrap.cc:492] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: Bootstrap starting.
I20260620 15:16:44.003170 4977 log.cc:824] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:44.004863 4979 tablet_bootstrap.cc:654] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:44.005553 4978 tablet_bootstrap.cc:654] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:44.006057 4979 log.cc:824] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:44.006230 4977 tablet_bootstrap.cc:492] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: No bootstrap required, opened a new log
I20260620 15:16:44.006489 4977 ts_tablet_manager.cc:1397] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 15:16:44.006556 4978 log.cc:824] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:44.008903 4979 tablet_bootstrap.cc:492] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: No bootstrap required, opened a new log
I20260620 15:16:44.009150 4979 ts_tablet_manager.cc:1397] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:16:44.010391 4978 tablet_bootstrap.cc:492] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: No bootstrap required, opened a new log
I20260620 15:16:44.010687 4978 ts_tablet_manager.cc:1397] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:16:44.017087 4978 raft_consensus.cc:348] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } }
I20260620 15:16:44.017095 4977 raft_consensus.cc:348] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } }
I20260620 15:16:44.017421 4978 raft_consensus.cc:374] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:44.017520 4977 raft_consensus.cc:374] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:44.017556 4978 raft_consensus.cc:729] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37daab7ada2846288060ce10faa63461, State: Initialized, Role: FOLLOWER
I20260620 15:16:44.017619 4977 raft_consensus.cc:729] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d04a5b16903747108483d8522bcae945, State: Initialized, Role: FOLLOWER
I20260620 15:16:44.018148 4978 consensus_queue.cc:260] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } }
I20260620 15:16:44.018265 4977 consensus_queue.cc:260] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } }
I20260620 15:16:44.019475 4827 heartbeater.cc:503] Master 127.0.97.126:43833 was elected leader, sending a full tablet report...
I20260620 15:16:44.019294 4979 raft_consensus.cc:348] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } }
I20260620 15:16:44.019685 4979 raft_consensus.cc:374] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:44.019775 4979 raft_consensus.cc:729] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183e0fdffa47437cab4382b372540294, State: Initialized, Role: FOLLOWER
I20260620 15:16:44.020107 4977 ts_tablet_manager.cc:1428] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 15:16:44.020531 4979 consensus_queue.cc:260] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } }
I20260620 15:16:44.023005 4979 ts_tablet_manager.cc:1428] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260620 15:16:44.024659 4959 heartbeater.cc:503] Master 127.0.97.126:43833 was elected leader, sending a full tablet report...
I20260620 15:16:44.025043 4978 ts_tablet_manager.cc:1428] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260620 15:16:44.139797 4960 tablet.cc:2369] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:16:44.163064 4983 raft_consensus.cc:482] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:44.163337 4983 raft_consensus.cc:504] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } }
I20260620 15:16:44.165023 4983 leader_election.cc:288] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d04a5b16903747108483d8522bcae945 (127.0.97.66:35141), 183e0fdffa47437cab4382b372540294 (127.0.97.65:37571)
W20260620 15:16:44.166029 4828 tablet.cc:2369] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:16:44.172006 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.165524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:35141 (local address 127.0.97.67:54625)
0620 15:16:44.165783 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:44.165809 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:44.165943 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:44.166469 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:44.166474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:44.166499 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:44.166686 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:44.166697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.167967 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.167971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:44.169868 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:44.169881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.171411 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.171414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:44.171548 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.171572 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:16:44.171642 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 15:16:44.171728 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 15:16:44.172212 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.165437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54625 (local address 127.0.97.66:35141)
0620 15:16:44.165632 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:44.165639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:44.165925 (+ 286us) server_negotiation.cc:407] Connection header received
0620 15:16:44.166131 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:44.166136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:44.166215 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:44.166297 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:44.166829 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.167829 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.170059 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.171262 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.171422 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.171533 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:44.171706 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 15:16:44.171861 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:16:44.172752 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.165545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35311 (local address 127.0.97.65:37571)
0620 15:16:44.165822 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:44.165830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:44.166503 (+ 673us) server_negotiation.cc:407] Connection header received
0620 15:16:44.166768 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:44.166774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:44.166851 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:44.166948 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:44.167395 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.168288 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.170469 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.171736 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.171881 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.172000 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:44.172355 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 15:16:44.172478 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260620 15:16:44.172828 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.165570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37571 (local address 127.0.97.67:35311)
0620 15:16:44.166398 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:44.166416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:44.166571 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:44.166965 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:44.166968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:44.166986 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:44.167250 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:44.167256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.168472 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.168478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:44.170320 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:44.170329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.171954 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.171959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:44.172145 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.172188 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 15:16:44.172327 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:16:44.172456 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":1,"thread_start_us":535,"threads_started":1}
I20260620 15:16:44.173331 4782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15710d89e46542d89823a9af0ab0b890" candidate_uuid: "37daab7ada2846288060ce10faa63461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04a5b16903747108483d8522bcae945" is_pre_election: true
I20260620 15:16:44.173888 4985 raft_consensus.cc:482] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:44.173943 4782 raft_consensus.cc:2454] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37daab7ada2846288060ce10faa63461 in term 0.
I20260620 15:16:44.173904 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15710d89e46542d89823a9af0ab0b890" candidate_uuid: "37daab7ada2846288060ce10faa63461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183e0fdffa47437cab4382b372540294" is_pre_election: true
I20260620 15:16:44.174239 4985 raft_consensus.cc:504] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } }
I20260620 15:16:44.174721 4849 leader_election.cc:302] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37daab7ada2846288060ce10faa63461, d04a5b16903747108483d8522bcae945; no voters:
I20260620 15:16:44.175082 4983 raft_consensus.cc:2781] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:44.175041 4649 raft_consensus.cc:2454] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37daab7ada2846288060ce10faa63461 in term 0.
I20260620 15:16:44.175220 4983 raft_consensus.cc:482] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:44.175311 4983 raft_consensus.cc:3037] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:44.176192 4985 leader_election.cc:288] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37daab7ada2846288060ce10faa63461 (127.0.97.67:38191), d04a5b16903747108483d8522bcae945 (127.0.97.66:35141)
I20260620 15:16:44.177875 4983 raft_consensus.cc:504] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } }
I20260620 15:16:44.178473 4983 leader_election.cc:288] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [CANDIDATE]: Term 1 election: Requested vote from peers d04a5b16903747108483d8522bcae945 (127.0.97.66:35141), 183e0fdffa47437cab4382b372540294 (127.0.97.65:37571)
I20260620 15:16:44.178763 4782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15710d89e46542d89823a9af0ab0b890" candidate_uuid: "37daab7ada2846288060ce10faa63461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04a5b16903747108483d8522bcae945"
I20260620 15:16:44.178946 4782 raft_consensus.cc:3037] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:44.181238 4782 raft_consensus.cc:2454] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37daab7ada2846288060ce10faa63461 in term 1.
I20260620 15:16:44.181504 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15710d89e46542d89823a9af0ab0b890" candidate_uuid: "37daab7ada2846288060ce10faa63461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "183e0fdffa47437cab4382b372540294"
I20260620 15:16:44.181728 4849 leader_election.cc:302] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37daab7ada2846288060ce10faa63461, d04a5b16903747108483d8522bcae945; no voters:
I20260620 15:16:44.181787 4649 raft_consensus.cc:3037] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:44.181993 4983 raft_consensus.cc:2781] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:44.186467 4649 raft_consensus.cc:2454] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37daab7ada2846288060ce10faa63461 in term 1.
I20260620 15:16:44.187271 4983 raft_consensus.cc:686] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 1 LEADER]: Becoming Leader. State: Replica: 37daab7ada2846288060ce10faa63461, State: Running, Role: LEADER
I20260620 15:16:44.188447 4983 consensus_queue.cc:237] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } }
I20260620 15:16:44.190029 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.176446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:38191 (local address 127.0.97.65:50031)
0620 15:16:44.181191 (+ 4745us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:44.181211 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:44.181386 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:44.181868 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:44.181875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:44.181899 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:44.182339 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:44.182356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.184204 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.184212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:44.187465 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:44.187479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.189305 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.189311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:44.189463 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.189496 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:16:44.189595 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:16:44.189724 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4604,"thread_start_us":11273,"threads_started":1}
I20260620 15:16:44.190029 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.176451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:50031 (local address 127.0.97.67:38191)
0620 15:16:44.176626 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:44.176631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:44.181289 (+ 4658us) server_negotiation.cc:407] Connection header received
0620 15:16:44.181545 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:44.181549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:44.181622 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:44.181680 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:44.182535 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.183986 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.187637 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.189113 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.189259 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.189371 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:44.189638 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 15:16:44.189763 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:16:44.191064 4914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15710d89e46542d89823a9af0ab0b890" candidate_uuid: "183e0fdffa47437cab4382b372540294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37daab7ada2846288060ce10faa63461" is_pre_election: true
I20260620 15:16:44.193876 4506 catalog_manager.cc:5495] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37daab7ada2846288060ce10faa63461 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "37daab7ada2846288060ce10faa63461" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37daab7ada2846288060ce10faa63461" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 } health_report { overall_health: UNKNOWN } } }
I20260620 15:16:44.194779 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.176513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:58715 (local address 127.0.97.66:35141)
0620 15:16:44.176684 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:44.176690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:44.188117 (+ 11427us) server_negotiation.cc:407] Connection header received
0620 15:16:44.188376 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:44.188384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:44.188469 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:44.188589 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:44.189160 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.190217 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.192836 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.193885 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.194010 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.194096 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:44.194423 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 15:16:44.194543 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:16:44.194808 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.176766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:35141 (local address 127.0.97.65:58715)
0620 15:16:44.188030 (+ 11264us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:44.188052 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:44.188215 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:44.188608 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:44.188613 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:44.188637 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:44.188972 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:44.188983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.190575 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.190579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:44.192679 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:44.192693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.194095 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.194100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:44.194210 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.194280 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 15:16:44.194389 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:16:44.194516 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11134,"thread_start_us":4075,"threads_started":1}
I20260620 15:16:44.195325 4782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15710d89e46542d89823a9af0ab0b890" candidate_uuid: "183e0fdffa47437cab4382b372540294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d04a5b16903747108483d8522bcae945" is_pre_election: true
I20260620 15:16:44.195618 4782 raft_consensus.cc:2379] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 183e0fdffa47437cab4382b372540294 in current term 1: Already voted for candidate 37daab7ada2846288060ce10faa63461 in this term.
I20260620 15:16:44.196239 4584 leader_election.cc:302] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 183e0fdffa47437cab4382b372540294; no voters: 37daab7ada2846288060ce10faa63461, d04a5b16903747108483d8522bcae945
I20260620 15:16:44.196650 4985 raft_consensus.cc:2726] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 15:16:44.196787 4695 tablet.cc:2369] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:16:44.216926 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.211305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.0.97.67:38191)
0620 15:16:44.211515 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:44.211522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:44.211675 (+ 153us) server_negotiation.cc:407] Connection header received
0620 15:16:44.211923 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:44.211927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:44.212046 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:44.212202 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:16:44.212619 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.213369 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.214753 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.215170 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.215310 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.215831 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:16:44.216290 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:16:44.216450 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:44.216551 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 15:16:44.216625 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:16:44.224779 4649 raft_consensus.cc:1261] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 1 FOLLOWER]: Refusing update from remote peer 37daab7ada2846288060ce10faa63461: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:44.224787 4782 raft_consensus.cc:1261] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [term 1 FOLLOWER]: Refusing update from remote peer 37daab7ada2846288060ce10faa63461: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:44.225747 4989 consensus_queue.cc:1035] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "183e0fdffa47437cab4382b372540294" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:44.226060 4983 consensus_queue.cc:1035] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d04a5b16903747108483d8522bcae945" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:44.240928 4992 mvcc.cc:205] Tried to move back new op lower bound from 7298943402893860864 to 7298943402760970240. Current Snapshot: MvccSnapshot[applied={T|T < 7298943402893860864 or (T in {7298943402893860864})}]
I20260620 15:16:44.250792 4505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:44.251106 4505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:44.254225 4505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07b06859d6db46c68c1662d58ee6dfb3: Sending DeleteTablet for 3 replicas of tablet 15710d89e46542d89823a9af0ab0b890
I20260620 15:16:44.255244 4894 tablet_service.cc:1514] Processing DeleteTablet for tablet 15710d89e46542d89823a9af0ab0b890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:44 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40260
I20260620 15:16:44.255393 4762 tablet_service.cc:1514] Processing DeleteTablet for tablet 15710d89e46542d89823a9af0ab0b890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:44 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:43470
I20260620 15:16:44.255569 4629 tablet_service.cc:1514] Processing DeleteTablet for tablet 15710d89e46542d89823a9af0ab0b890 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:44 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40124
I20260620 15:16:44.257396 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 4567
I20260620 15:16:44.266741 5004 tablet_replica.cc:330] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: stopping tablet replica
I20260620 15:16:44.267383 5004 raft_consensus.cc:2229] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:44.268074 5003 tablet_replica.cc:330] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: stopping tablet replica
I20260620 15:16:44.268671 5004 raft_consensus.cc:2258] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:44.269201 5003 raft_consensus.cc:2229] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:44.270313 5003 raft_consensus.cc:2258] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:44.272531 5004 ts_tablet_manager.cc:1905] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:44.273305 5002 tablet_replica.cc:330] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: stopping tablet replica
I20260620 15:16:44.274914 5003 ts_tablet_manager.cc:1905] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:44.275326 5002 raft_consensus.cc:2229] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:16:44.275943 5002 raft_consensus.cc:2258] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 15:16:44 dist-test-slave-7s64 krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:44.277880 5002 ts_tablet_manager.cc:1905] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:44.280011 5004 ts_tablet_manager.cc:1918] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:16:44.280221 5004 log.cc:1196] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/15710d89e46542d89823a9af0ab0b890
I20260620 15:16:44.280792 5004 ts_tablet_manager.cc:1939] T 15710d89e46542d89823a9af0ab0b890 P 183e0fdffa47437cab4382b372540294: Deleting consensus metadata
I20260620 15:16:44.282541 4493 catalog_manager.cc:4841] TS 183e0fdffa47437cab4382b372540294 (127.0.97.65:37571): tablet 15710d89e46542d89823a9af0ab0b890 (table test-table [id=8d99c4e9cc774162ab9394f6a427ff6b]) successfully deleted
I20260620 15:16:44.285738 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.259202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.0.97.65:37571)
0620 15:16:44.259382 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:44.259389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:44.264675 (+ 5286us) server_negotiation.cc:407] Connection header received
0620 15:16:44.264878 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:44.264883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:44.264985 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:44.265083 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:44.269417 (+ 4334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.270613 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.272456 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.273113 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.273207 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.278337 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:44.278349 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:44.278372 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:44.278420 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:44.282316 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:44.283250 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:44.283261 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:44.283266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:44.283430 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:44.283865 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:44.283870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:44.283874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:44.284294 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:44.284614 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:44.285132 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 15:16:44.285347 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:16:44.287233 4689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:44.287386 5003 ts_tablet_manager.cc:1918] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:16:44.287447 5002 ts_tablet_manager.cc:1918] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:16:44.287593 5002 log.cc:1196] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/15710d89e46542d89823a9af0ab0b890
I20260620 15:16:44.287619 5003 log.cc:1196] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/15710d89e46542d89823a9af0ab0b890
I20260620 15:16:44.288174 5003 ts_tablet_manager.cc:1939] T 15710d89e46542d89823a9af0ab0b890 P d04a5b16903747108483d8522bcae945: Deleting consensus metadata
I20260620 15:16:44.288192 5002 ts_tablet_manager.cc:1939] T 15710d89e46542d89823a9af0ab0b890 P 37daab7ada2846288060ce10faa63461: Deleting consensus metadata
I20260620 15:16:44.289738 4493 catalog_manager.cc:4841] TS 37daab7ada2846288060ce10faa63461 (127.0.97.67:38191): tablet 15710d89e46542d89823a9af0ab0b890 (table test-table [id=8d99c4e9cc774162ab9394f6a427ff6b]) successfully deleted
I20260620 15:16:44.289772 4492 catalog_manager.cc:4841] TS d04a5b16903747108483d8522bcae945 (127.0.97.66:35141): tablet 15710d89e46542d89823a9af0ab0b890 (table test-table [id=8d99c4e9cc774162ab9394f6a427ff6b]) successfully deleted
I20260620 15:16:44.378633 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 4567
I20260620 15:16:44.396530 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 4700
Jun 20 15:16:44 dist-test-slave-7s64 krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:16:44.411134 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.397249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.0.97.66:35141)
0620 15:16:44.397433 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:44.397440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:44.397522 (+ 82us) server_negotiation.cc:407] Connection header received
0620 15:16:44.397728 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:44.397739 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:44.397802 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:44.397911 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:44.399095 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.399852 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.401249 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.401633 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.401761 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.405581 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:44.405594 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:44.405610 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:44.405641 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:44.408353 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:44.409149 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:44.409159 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:44.409162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:44.409300 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:44.409677 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:44.409688 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:44.409691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:44.410012 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:44.410320 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:44.410582 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 15:16:44.410800 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:16:44.411782 4822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:44.515751 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 4700
I20260620 15:16:44.532656 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 4832
Jun 20 15:16:44 dist-test-slave-7s64 krb5kdc[4462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:16:44.550487 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:44.533549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.0.97.67:38191)
0620 15:16:44.533748 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:44.533755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:44.533777 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:16:44.533965 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:44.533973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:44.534068 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:44.534189 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:44.535387 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.536645 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.538117 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:44.538806 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:44.538944 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:44.543459 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:44.543476 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:44.543496 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:44.543539 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:44.547657 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:44.548248 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:44.548270 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:44.548274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:44.548423 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:44.548848 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:44.548853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:44.548856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:44.549374 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:44.549661 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:44.549898 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 15:16:44.550114 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:16:44.551188 4954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:44.668807 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 4832
I20260620 15:16:44.687600 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 4475
I20260620 15:16:44.688738 4536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:44.793579 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 4475
2026-06-20T15:16:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4065 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T15:16:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:44Z Disabled control of system clock
I20260620 15:16:44.857908 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33907
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:33907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 15:16:45.168771 5013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:45.169042 5013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:45.169123 5013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:45.176169 5013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:45.176268 5013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:45.176304 5013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:45.176352 5013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:45.176394 5013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:45.185312 5013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:33907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:45.186440 5013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:45.188073 5013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:45.199177 5021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:45.199227 5018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:45.199254 5019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:45.200102 5013 server_base.cc:1034] running on GCE node
I20260620 15:16:45.200879 5013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:45.202114 5013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:45.203332 5013 hybrid_clock.cc:648] HybridClock initialized: now 1781968605203291 us; error 56 us; skew 500 ppm
I20260620 15:16:45.203650 5013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:16:45.212699 5013 webserver.cc:460] Webserver started at http://127.0.97.126:36383/ using document root <none> and password file <none>
I20260620 15:16:45.213372 5013 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:45.213508 5013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:45.213847 5013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:45.216600 5013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "00f68014954f479caf49feb7bc846a9a"
format_stamp: "Formatted at 2026-06-20 15:16:45 on dist-test-slave-7s64"
server_key: "3251d3275c38120c15526ddf49444bcf"
server_key_iv: "021fa7b918cdae7443d755246493cdea"
server_key_version: "encryptionkey@0"
I20260620 15:16:45.217373 5013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "00f68014954f479caf49feb7bc846a9a"
format_stamp: "Formatted at 2026-06-20 15:16:45 on dist-test-slave-7s64"
server_key: "3251d3275c38120c15526ddf49444bcf"
server_key_iv: "021fa7b918cdae7443d755246493cdea"
server_key_version: "encryptionkey@0"
I20260620 15:16:45.222138 5013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:16:45.225569 5027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:45.226861 5013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:16:45.227038 5013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "00f68014954f479caf49feb7bc846a9a"
format_stamp: "Formatted at 2026-06-20 15:16:45 on dist-test-slave-7s64"
server_key: "3251d3275c38120c15526ddf49444bcf"
server_key_iv: "021fa7b918cdae7443d755246493cdea"
server_key_version: "encryptionkey@0"
I20260620 15:16:45.227240 5013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:45.232805 5013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:45.233503 5013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:45.233781 5013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:45.251016 5013 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:33907
I20260620 15:16:45.251091 5078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:33907 every 8 connection(s)
I20260620 15:16:45.252197 5013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:45.253741 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 5013
I20260620 15:16:45.254007 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:45.254822 389 external_mini_cluster.cc:1428] Setting key 187bf90d761238263f7847f5636e61e5
I20260620 15:16:45.256875 5079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:45.262717 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:45.256897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.0.97.126:33907)
0620 15:16:45.257880 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:45.257889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:45.257927 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:16:45.259345 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:45.259365 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:45.259744 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:45.260053 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:45.261225 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:45.261231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:45.261242 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:45.261258 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:45.261329 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:45.261352 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:45.261506 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:45.261609 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 15:16:45.261856 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":378,"threads_started":1}
I20260620 15:16:45.268855 5079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a: Bootstrap starting.
I20260620 15:16:45.271718 5079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:45.272768 5079 log.cc:824] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:45.275265 5079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a: No bootstrap required, opened a new log
I20260620 15:16:45.282624 5079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f68014954f479caf49feb7bc846a9a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33907 } }
I20260620 15:16:45.282967 5079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:45.283092 5079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f68014954f479caf49feb7bc846a9a, State: Initialized, Role: FOLLOWER
I20260620 15:16:45.283646 5079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f68014954f479caf49feb7bc846a9a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33907 } }
I20260620 15:16:45.283874 5079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:45.284005 5079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:45.284189 5079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:45.286372 5079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f68014954f479caf49feb7bc846a9a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33907 } }
I20260620 15:16:45.286926 5079 leader_election.cc:302] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00f68014954f479caf49feb7bc846a9a; no voters:
I20260620 15:16:45.287478 5079 leader_election.cc:288] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:45.287631 5084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:45.288784 5084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [term 1 LEADER]: Becoming Leader. State: Replica: 00f68014954f479caf49feb7bc846a9a, State: Running, Role: LEADER
I20260620 15:16:45.289317 5084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f68014954f479caf49feb7bc846a9a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33907 } }
I20260620 15:16:45.289605 5079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:45.292276 5085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00f68014954f479caf49feb7bc846a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f68014954f479caf49feb7bc846a9a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33907 } } }
I20260620 15:16:45.292757 5085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:45.292275 5086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00f68014954f479caf49feb7bc846a9a. Latest consensus state: current_term: 1 leader_uuid: "00f68014954f479caf49feb7bc846a9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f68014954f479caf49feb7bc846a9a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33907 } } }
I20260620 15:16:45.293337 5086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:45.294222 5092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:45.299238 5092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:45.307651 5092 catalog_manager.cc:1347] Generated new cluster ID: 228b04e64077470c9f9def961c569c6e
I20260620 15:16:45.307770 5092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:45.327221 5092 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:45.327608 5092 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:45.338932 5092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a: Generated new TSK 0
I20260620 15:16:45.339670 5092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:16:45.346341 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33907
--builtin_ntp_servers=127.0.97.84:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 15:16:45.663160 5103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:45.663398 5103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:45.663542 5103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:45.670362 5103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:45.670447 5103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:45.670610 5103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:45.679081 5103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:45.680235 5103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:45.681919 5103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:45.694837 5111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:45.694885 5109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:45.694885 5108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:45.695843 5103 server_base.cc:1034] running on GCE node
I20260620 15:16:45.696434 5103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:45.697304 5103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:45.698489 5103 hybrid_clock.cc:648] HybridClock initialized: now 1781968605698456 us; error 25 us; skew 500 ppm
I20260620 15:16:45.698773 5103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:16:45.700876 5103 webserver.cc:460] Webserver started at http://127.0.97.65:37975/ using document root <none> and password file <none>
I20260620 15:16:45.701475 5103 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:45.701583 5103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:45.701890 5103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:45.704438 5103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "9c57c9ff2c314f14b85993e52fc88570"
format_stamp: "Formatted at 2026-06-20 15:16:45 on dist-test-slave-7s64"
server_key: "3f5cb018fa9f4beb8e9021ac6c5eaaac"
server_key_iv: "334239138fd46cf60e17e12458b73958"
server_key_version: "encryptionkey@0"
I20260620 15:16:45.705257 5103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "9c57c9ff2c314f14b85993e52fc88570"
format_stamp: "Formatted at 2026-06-20 15:16:45 on dist-test-slave-7s64"
server_key: "3f5cb018fa9f4beb8e9021ac6c5eaaac"
server_key_iv: "334239138fd46cf60e17e12458b73958"
server_key_version: "encryptionkey@0"
I20260620 15:16:45.709878 5103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:16:45.713096 5117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:45.714309 5103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:45.714473 5103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "9c57c9ff2c314f14b85993e52fc88570"
format_stamp: "Formatted at 2026-06-20 15:16:45 on dist-test-slave-7s64"
server_key: "3f5cb018fa9f4beb8e9021ac6c5eaaac"
server_key_iv: "334239138fd46cf60e17e12458b73958"
server_key_version: "encryptionkey@0"
I20260620 15:16:45.714684 5103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:45.720080 5103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:45.721040 5103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:45.721299 5103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:45.722141 5103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:45.723459 5103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:45.723572 5103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:45.723706 5103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:45.723793 5103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:45.755832 5103 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:34921
I20260620 15:16:45.755911 5229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:34921 every 8 connection(s)
I20260620 15:16:45.757041 5103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:45.763135 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 5103
I20260620 15:16:45.763345 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:45.764015 389 external_mini_cluster.cc:1428] Setting key 15769a32d0b561c1a4ba0b8646748086
I20260620 15:16:45.765446 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:45.759634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:40017 (local address 127.0.97.126:33907)
0620 15:16:45.759836 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:45.759841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:45.761186 (+ 1345us) server_negotiation.cc:407] Connection header received
0620 15:16:45.762948 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:45.762955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:45.763031 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:45.763165 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:45.764640 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:45.764645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:45.764648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:45.764664 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:45.764693 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:45.764704 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:45.764844 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:45.765065 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 15:16:45.765133 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:16:45.766593 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:45.760103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33907 (local address 127.0.97.65:40017)
0620 15:16:45.761026 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:45.761070 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:45.762675 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:45.763394 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:45.763402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:45.763534 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:45.764302 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:16:45.764305 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:45.764342 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:16:45.764422 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:45.764852 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:45.764860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:45.764866 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:16:45.765143 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 15:16:45.765384 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":257,"threads_started":1}
I20260620 15:16:45.766960 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33907
--builtin_ntp_servers=127.0.97.84:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 15:16:45.767910 5230 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33907
I20260620 15:16:45.768193 5230 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:45.768752 5230 heartbeater.cc:511] Master 127.0.97.126:33907 requested a full tablet report, sending...
I20260620 15:16:45.770696 5044 ts_manager.cc:194] Registered new tserver with Master: 9c57c9ff2c314f14b85993e52fc88570 (127.0.97.65:34921)
W20260620 15:16:46.073267 5233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:46.073524 5233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:46.073611 5233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:46.080363 5233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:46.080453 5233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:46.080628 5233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:46.089607 5233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:46.090718 5233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:46.092362 5233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:46.105175 5241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:46.105306 5239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:46.105264 5238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:46.105908 5233 server_base.cc:1034] running on GCE node
I20260620 15:16:46.106444 5233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:46.107254 5233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:46.108479 5233 hybrid_clock.cc:648] HybridClock initialized: now 1781968606108443 us; error 36 us; skew 500 ppm
I20260620 15:16:46.108829 5233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:16:46.110992 5233 webserver.cc:460] Webserver started at http://127.0.97.66:46559/ using document root <none> and password file <none>
I20260620 15:16:46.111644 5233 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:46.111776 5233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:46.112077 5233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:46.114759 5233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b"
format_stamp: "Formatted at 2026-06-20 15:16:46 on dist-test-slave-7s64"
server_key: "488d04cead9e48dd6d4367104b30a3cb"
server_key_iv: "1a97df3ad2da9a8383be871faa58931f"
server_key_version: "encryptionkey@0"
I20260620 15:16:46.115514 5233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b"
format_stamp: "Formatted at 2026-06-20 15:16:46 on dist-test-slave-7s64"
server_key: "488d04cead9e48dd6d4367104b30a3cb"
server_key_iv: "1a97df3ad2da9a8383be871faa58931f"
server_key_version: "encryptionkey@0"
I20260620 15:16:46.120314 5233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 15:16:46.123497 5247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:46.124742 5233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:46.124903 5233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b"
format_stamp: "Formatted at 2026-06-20 15:16:46 on dist-test-slave-7s64"
server_key: "488d04cead9e48dd6d4367104b30a3cb"
server_key_iv: "1a97df3ad2da9a8383be871faa58931f"
server_key_version: "encryptionkey@0"
I20260620 15:16:46.125118 5233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:46.130393 5233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:46.131124 5233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:46.131374 5233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:46.132241 5233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:46.133643 5233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:46.133757 5233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:46.133884 5233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:46.133981 5233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:46.166321 5233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:38595
I20260620 15:16:46.166374 5359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:38595 every 8 connection(s)
I20260620 15:16:46.167519 5233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:46.174155 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 5233
I20260620 15:16:46.174345 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:46.174782 389 external_mini_cluster.cc:1428] Setting key 62a72ee487b462f747694d3a611a89e1
I20260620 15:16:46.175971 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.170177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:35641 (local address 127.0.97.126:33907)
0620 15:16:46.170398 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:46.170403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:46.171744 (+ 1341us) server_negotiation.cc:407] Connection header received
0620 15:16:46.173571 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:46.173576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:46.173625 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:46.173722 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:46.175195 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:46.175201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:46.175205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:46.175223 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:46.175263 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:46.175280 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:46.175400 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:46.175631 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 15:16:46.175699 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 15:16:46.177165 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.170640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33907 (local address 127.0.97.66:35641)
0620 15:16:46.171582 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:46.171625 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:46.173309 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:46.173943 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:46.173951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:46.174043 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:46.174845 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:16:46.174848 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:46.174920 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:16:46.175022 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:46.175397 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:46.175408 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:46.175416 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:16:46.175703 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 15:16:46.175995 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":284,"threads_started":1}
I20260620 15:16:46.177917 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33907
--builtin_ntp_servers=127.0.97.84:35311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 15:16:46.178474 5360 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33907
I20260620 15:16:46.178776 5360 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:46.179409 5360 heartbeater.cc:511] Master 127.0.97.126:33907 requested a full tablet report, sending...
I20260620 15:16:46.180908 5044 ts_manager.cc:194] Registered new tserver with Master: a1fdadbf15e54aaf81fa1045a3a6c38b (127.0.97.66:38595)
W20260620 15:16:46.494071 5363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:46.494309 5363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:46.494450 5363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:46.501639 5363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:46.501722 5363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:46.501881 5363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:46.510838 5363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:46.512426 5363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:46.514387 5363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:46.527187 5371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:46.527249 5368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:46.527251 5369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:46.528262 5363 server_base.cc:1034] running on GCE node
I20260620 15:16:46.528822 5363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:46.529764 5363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:46.530943 5363 hybrid_clock.cc:648] HybridClock initialized: now 1781968606530902 us; error 41 us; skew 500 ppm
I20260620 15:16:46.531276 5363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:16:46.533396 5363 webserver.cc:460] Webserver started at http://127.0.97.67:42721/ using document root <none> and password file <none>
I20260620 15:16:46.533998 5363 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:46.534102 5363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:46.534400 5363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:46.537261 5363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "d3e1bd59bc244a5791068bd335e5cb65"
format_stamp: "Formatted at 2026-06-20 15:16:46 on dist-test-slave-7s64"
server_key: "9592c9e574de0ee3bd3dac3de1aea0ea"
server_key_iv: "cad3728c43d71444ee635410b3917599"
server_key_version: "encryptionkey@0"
I20260620 15:16:46.538003 5363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "d3e1bd59bc244a5791068bd335e5cb65"
format_stamp: "Formatted at 2026-06-20 15:16:46 on dist-test-slave-7s64"
server_key: "9592c9e574de0ee3bd3dac3de1aea0ea"
server_key_iv: "cad3728c43d71444ee635410b3917599"
server_key_version: "encryptionkey@0"
I20260620 15:16:46.542745 5363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:16:46.545969 5377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:46.547072 5363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:16:46.547221 5363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "d3e1bd59bc244a5791068bd335e5cb65"
format_stamp: "Formatted at 2026-06-20 15:16:46 on dist-test-slave-7s64"
server_key: "9592c9e574de0ee3bd3dac3de1aea0ea"
server_key_iv: "cad3728c43d71444ee635410b3917599"
server_key_version: "encryptionkey@0"
I20260620 15:16:46.547433 5363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:46.552815 5363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:46.553527 5363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:46.553776 5363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:46.554616 5363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:46.555902 5363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:46.555984 5363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:46.556128 5363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:46.556217 5363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:46.587767 5363 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:43865
I20260620 15:16:46.587819 5489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:43865 every 8 connection(s)
I20260620 15:16:46.589078 5363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:46.596936 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 5363
I20260620 15:16:46.597182 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:46.597230 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.591680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54253 (local address 127.0.97.126:33907)
0620 15:16:46.591853 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:46.591857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:46.593240 (+ 1383us) server_negotiation.cc:407] Connection header received
0620 15:16:46.595075 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:46.595082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:46.595148 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:46.595258 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:46.596525 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:46.596531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:46.596536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:46.596554 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:46.596594 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:46.596605 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:46.596735 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:46.596919 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 15:16:46.596981 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:16:46.597647 389 external_mini_cluster.cc:1428] Setting key bfb8e3cf5ef424c997178617cb848ac0
I20260620 15:16:46.598171 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.592195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33907 (local address 127.0.97.67:54253)
0620 15:16:46.593084 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:46.593122 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:46.594797 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:46.595474 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:46.595482 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:46.595565 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:46.596226 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:16:46.596229 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:46.596263 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:16:46.596341 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:46.596698 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:46.596707 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:46.596716 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 15:16:46.596991 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 15:16:46.597221 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":298,"threads_started":1}
I20260620 15:16:46.599433 5490 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33907
I20260620 15:16:46.599730 5490 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:46.600340 5490 heartbeater.cc:511] Master 127.0.97.126:33907 requested a full tablet report, sending...
I20260620 15:16:46.601637 5044 ts_manager.cc:194] Registered new tserver with Master: d3e1bd59bc244a5791068bd335e5cb65 (127.0.97.67:43865)
I20260620 15:16:46.612565 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:16:46.619212 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.616834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.0.97.126:33907)
0620 15:16:46.617003 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:46.617010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:46.617527 (+ 517us) server_negotiation.cc:407] Connection header received
0620 15:16:46.617661 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:46.617665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:46.617707 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:46.617799 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:46.618635 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:46.618639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:46.618642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:46.618655 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:46.618682 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:46.618716 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:46.618830 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:46.618867 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 15:16:46.618936 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:16:46.622582 5044 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:16:46.629170 5044 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:16:46.645639 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.640528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.0.97.65:34921)
0620 15:16:46.641995 (+ 1467us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:46.642006 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:46.642911 (+ 905us) server_negotiation.cc:407] Connection header received
0620 15:16:46.643147 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:46.643155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:46.643337 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:46.643453 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:46.644799 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:46.644809 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:46.644824 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:46.644843 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:46.644897 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:46.644926 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:46.645055 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:46.645238 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 15:16:46.645337 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1311,"thread_start_us":1196,"threads_started":1}
I20260620 15:16:46.645740 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.639791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:43865 (local address 127.0.0.1:41458)
0620 15:16:46.642338 (+ 2547us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:46.642387 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:46.642547 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:46.643087 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:46.643095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:46.643177 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:46.644265 (+ 1088us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:16:46.644271 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:46.644301 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:16:46.644383 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:46.645102 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:46.645106 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:46.645110 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 15:16:46.645269 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 15:16:46.645360 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"thread_start_us":347,"threads_started":1}
I20260620 15:16:46.645792 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.639980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.0.97.67:43865)
0620 15:16:46.641570 (+ 1590us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:46.641578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:46.642525 (+ 947us) server_negotiation.cc:407] Connection header received
0620 15:16:46.642670 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:46.642679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:46.642903 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:46.643042 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:46.644563 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:46.644571 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:46.644584 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:46.644604 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:46.644653 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:46.644667 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:46.644854 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:46.645276 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 15:16:46.645374 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1417,"thread_start_us":1267,"threads_started":1}
I20260620 15:16:46.646562 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.640002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34921 (local address 127.0.0.1:52594)
0620 15:16:46.642793 (+ 2791us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:46.642822 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:46.643003 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:46.643798 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:46.643805 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:46.643831 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:46.644604 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:16:46.644607 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:46.644623 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:16:46.644659 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:46.645037 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:46.645046 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:46.645053 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:16:46.646087 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0620 15:16:46.646123 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2658,"thread_start_us":2823,"threads_started":1}
I20260620 15:16:46.647300 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.641828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46930 (local address 127.0.97.66:38595)
0620 15:16:46.642375 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:46.642380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:46.643169 (+ 789us) server_negotiation.cc:407] Connection header received
0620 15:16:46.644444 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:46.644454 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:46.644669 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:46.644777 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:46.646533 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:46.646548 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:46.646558 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:46.646578 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:46.646644 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:46.646666 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:46.646830 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:46.646895 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 15:16:46.646974 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":298,"threads_started":1}
I20260620 15:16:46.647351 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.641238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:38595 (local address 127.0.0.1:46930)
0620 15:16:46.643077 (+ 1839us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:46.643092 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:46.643252 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:46.644863 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:46.644869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:46.644889 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:46.646310 (+ 1421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:16:46.646313 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:46.646328 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:16:46.646370 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:46.646752 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:46.646757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:46.646762 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:16:46.646877 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 15:16:46.646921 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1729,"thread_start_us":1295,"threads_started":1}
I20260620 15:16:46.649657 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 6a447a43e587406ba029defe7dc2d876 (DEFAULT_TABLE table=test-table [id=346ee000f726478f9f0320a6f53324e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:46.650966 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a447a43e587406ba029defe7dc2d876. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:46.650463 5165 tablet_service.cc:1467] Processing CreateTablet for tablet 6a447a43e587406ba029defe7dc2d876 (DEFAULT_TABLE table=test-table [id=346ee000f726478f9f0320a6f53324e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:46.651727 5165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a447a43e587406ba029defe7dc2d876. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:46.653915 5295 tablet_service.cc:1467] Processing CreateTablet for tablet 6a447a43e587406ba029defe7dc2d876 (DEFAULT_TABLE table=test-table [id=346ee000f726478f9f0320a6f53324e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:16:46.655112 5295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a447a43e587406ba029defe7dc2d876. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:46.661398 5508 tablet_bootstrap.cc:492] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: Bootstrap starting.
I20260620 15:16:46.665081 5508 tablet_bootstrap.cc:654] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:46.665244 5509 tablet_bootstrap.cc:492] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570: Bootstrap starting.
I20260620 15:16:46.666421 5508 log.cc:824] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:46.668782 5511 tablet_bootstrap.cc:492] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: Bootstrap starting.
I20260620 15:16:46.668845 5509 tablet_bootstrap.cc:654] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:46.669554 5508 tablet_bootstrap.cc:492] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: No bootstrap required, opened a new log
I20260620 15:16:46.669824 5508 ts_tablet_manager.cc:1397] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 15:16:46.670123 5509 log.cc:824] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:46.672063 5511 tablet_bootstrap.cc:654] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:46.673029 5509 tablet_bootstrap.cc:492] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570: No bootstrap required, opened a new log
I20260620 15:16:46.673062 5511 log.cc:824] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:46.673264 5509 ts_tablet_manager.cc:1397] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:16:46.675359 5511 tablet_bootstrap.cc:492] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: No bootstrap required, opened a new log
I20260620 15:16:46.675599 5511 ts_tablet_manager.cc:1397] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 15:16:46.679767 5508 raft_consensus.cc:348] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e1bd59bc244a5791068bd335e5cb65" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43865 } } peers { permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 } } peers { permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 } }
I20260620 15:16:46.680160 5508 raft_consensus.cc:374] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:46.680254 5508 raft_consensus.cc:729] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e1bd59bc244a5791068bd335e5cb65, State: Initialized, Role: FOLLOWER
I20260620 15:16:46.680853 5508 consensus_queue.cc:260] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e1bd59bc244a5791068bd335e5cb65" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43865 } } peers { permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 } } peers { permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 } }
I20260620 15:16:46.681886 5490 heartbeater.cc:503] Master 127.0.97.126:33907 was elected leader, sending a full tablet report...
I20260620 15:16:46.682482 5508 ts_tablet_manager.cc:1428] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 15:16:46.683476 5511 raft_consensus.cc:348] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e1bd59bc244a5791068bd335e5cb65" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43865 } } peers { permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 } } peers { permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 } }
I20260620 15:16:46.683543 5509 raft_consensus.cc:348] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e1bd59bc244a5791068bd335e5cb65" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43865 } } peers { permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 } } peers { permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 } }
I20260620 15:16:46.683781 5511 raft_consensus.cc:374] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:46.683911 5511 raft_consensus.cc:729] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fdadbf15e54aaf81fa1045a3a6c38b, State: Initialized, Role: FOLLOWER
I20260620 15:16:46.683943 5509 raft_consensus.cc:374] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:46.684031 5509 raft_consensus.cc:729] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c57c9ff2c314f14b85993e52fc88570, State: Initialized, Role: FOLLOWER
I20260620 15:16:46.684597 5511 consensus_queue.cc:260] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e1bd59bc244a5791068bd335e5cb65" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43865 } } peers { permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 } } peers { permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 } }
I20260620 15:16:46.684595 5509 consensus_queue.cc:260] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e1bd59bc244a5791068bd335e5cb65" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43865 } } peers { permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 } } peers { permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 } }
I20260620 15:16:46.685518 5360 heartbeater.cc:503] Master 127.0.97.126:33907 was elected leader, sending a full tablet report...
I20260620 15:16:46.686115 5511 ts_tablet_manager.cc:1428] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260620 15:16:46.686187 5230 heartbeater.cc:503] Master 127.0.97.126:33907 was elected leader, sending a full tablet report...
I20260620 15:16:46.687327 5509 ts_tablet_manager.cc:1428] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260620 15:16:46.759194 5231 tablet.cc:2369] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:16:46.776368 5514 raft_consensus.cc:482] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:46.776661 5514 raft_consensus.cc:504] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e1bd59bc244a5791068bd335e5cb65" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43865 } } peers { permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 } } peers { permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 } }
I20260620 15:16:46.778102 5514 leader_election.cc:288] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c57c9ff2c314f14b85993e52fc88570 (127.0.97.65:34921), a1fdadbf15e54aaf81fa1045a3a6c38b (127.0.97.66:38595)
I20260620 15:16:46.780890 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.778340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34921 (local address 127.0.97.67:59343)
0620 15:16:46.778531 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:46.778551 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:46.778661 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:46.779213 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:46.779217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:46.779236 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:46.780015 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:16:46.780017 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:46.780033 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:16:46.780070 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:46.780453 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:46.780457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:46.780460 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 15:16:46.780604 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 15:16:46.780642 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 15:16:46.780947 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.778740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59343 (local address 127.0.97.65:34921)
0620 15:16:46.778936 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:46.778944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:46.778963 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:16:46.779016 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:46.779022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:46.779076 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:46.779177 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:46.780230 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:46.780235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:46.780238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:46.780254 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:46.780292 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:46.780306 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:46.780483 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:46.780631 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 15:16:46.780697 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:16:46.781610 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.778547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:46049 (local address 127.0.97.66:38595)
0620 15:16:46.778716 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:46.778724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:46.779289 (+ 565us) server_negotiation.cc:407] Connection header received
0620 15:16:46.779423 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:46.779431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:46.779493 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:46.779610 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:46.780767 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:46.780773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:46.780778 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:46.780795 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:46.780827 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:46.780841 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:46.780973 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:46.781262 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 15:16:46.781339 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:16:46.781793 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.778444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:38595 (local address 127.0.97.67:46049)
0620 15:16:46.779178 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:46.779199 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:46.779296 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:46.779710 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:46.779716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:46.779753 (+ 37us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:46.780546 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:16:46.780549 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:46.780567 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:16:46.780613 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:46.781115 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:46.781123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:46.781131 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:16:46.781273 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 15:16:46.781322 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":409,"threads_started":1}
I20260620 15:16:46.781790 5185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a447a43e587406ba029defe7dc2d876" candidate_uuid: "d3e1bd59bc244a5791068bd335e5cb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c57c9ff2c314f14b85993e52fc88570" is_pre_election: true
I20260620 15:16:46.782388 5185 raft_consensus.cc:2454] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3e1bd59bc244a5791068bd335e5cb65 in term 0.
I20260620 15:16:46.782734 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a447a43e587406ba029defe7dc2d876" candidate_uuid: "d3e1bd59bc244a5791068bd335e5cb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" is_pre_election: true
I20260620 15:16:46.783025 5378 leader_election.cc:302] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c57c9ff2c314f14b85993e52fc88570, d3e1bd59bc244a5791068bd335e5cb65; no voters:
I20260620 15:16:46.783206 5315 raft_consensus.cc:2454] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3e1bd59bc244a5791068bd335e5cb65 in term 0.
I20260620 15:16:46.783391 5514 raft_consensus.cc:2781] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:16:46.783540 5514 raft_consensus.cc:482] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:16:46.783664 5514 raft_consensus.cc:3037] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:46.786022 5514 raft_consensus.cc:504] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e1bd59bc244a5791068bd335e5cb65" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43865 } } peers { permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 } } peers { permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 } }
I20260620 15:16:46.786623 5514 leader_election.cc:288] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [CANDIDATE]: Term 1 election: Requested vote from peers 9c57c9ff2c314f14b85993e52fc88570 (127.0.97.65:34921), a1fdadbf15e54aaf81fa1045a3a6c38b (127.0.97.66:38595)
I20260620 15:16:46.786883 5185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a447a43e587406ba029defe7dc2d876" candidate_uuid: "d3e1bd59bc244a5791068bd335e5cb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c57c9ff2c314f14b85993e52fc88570"
I20260620 15:16:46.787061 5185 raft_consensus.cc:3037] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:46.787092 5315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a447a43e587406ba029defe7dc2d876" candidate_uuid: "d3e1bd59bc244a5791068bd335e5cb65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b"
I20260620 15:16:46.787338 5315 raft_consensus.cc:3037] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:46.789302 5185 raft_consensus.cc:2454] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3e1bd59bc244a5791068bd335e5cb65 in term 1.
I20260620 15:16:46.789726 5378 leader_election.cc:302] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c57c9ff2c314f14b85993e52fc88570, d3e1bd59bc244a5791068bd335e5cb65; no voters:
I20260620 15:16:46.789875 5315 raft_consensus.cc:2454] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3e1bd59bc244a5791068bd335e5cb65 in term 1.
I20260620 15:16:46.789978 5514 raft_consensus.cc:2781] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:46.790597 5514 raft_consensus.cc:686] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 1 LEADER]: Becoming Leader. State: Replica: d3e1bd59bc244a5791068bd335e5cb65, State: Running, Role: LEADER
I20260620 15:16:46.791121 5514 consensus_queue.cc:237] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e1bd59bc244a5791068bd335e5cb65" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43865 } } peers { permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 } } peers { permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 } }
I20260620 15:16:46.795722 5044 catalog_manager.cc:5495] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3e1bd59bc244a5791068bd335e5cb65 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "d3e1bd59bc244a5791068bd335e5cb65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3e1bd59bc244a5791068bd335e5cb65" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 } health_report { overall_health: UNKNOWN } } }
I20260620 15:16:46.835466 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.833523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41474 (local address 127.0.97.67:43865)
0620 15:16:46.833689 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:46.833693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:46.833782 (+ 89us) server_negotiation.cc:407] Connection header received
0620 15:16:46.833977 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:46.833980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:46.834054 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:46.834139 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:46.834943 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:46.834947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:46.834950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:46.834963 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:46.834991 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:46.835001 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:46.835095 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:46.835240 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 15:16:46.835281 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:16:46.840758 5491 tablet.cc:2369] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:16:46.843288 5315 raft_consensus.cc:1261] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b [term 1 FOLLOWER]: Refusing update from remote peer d3e1bd59bc244a5791068bd335e5cb65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:46.843396 5185 raft_consensus.cc:1261] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570 [term 1 FOLLOWER]: Refusing update from remote peer d3e1bd59bc244a5791068bd335e5cb65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:16:46.844234 5518 consensus_queue.cc:1035] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1fdadbf15e54aaf81fa1045a3a6c38b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:46.844628 5514 consensus_queue.cc:1035] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c57c9ff2c314f14b85993e52fc88570" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:46.856114 5521 mvcc.cc:205] Tried to move back new op lower bound from 7298943413621735424 to 7298943413420539904. Current Snapshot: MvccSnapshot[applied={T|T < 7298943413621735424}]
I20260620 15:16:46.872098 5042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:46.872376 5042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:16:46.875550 5042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00f68014954f479caf49feb7bc846a9a: Sending DeleteTablet for 3 replicas of tablet 6a447a43e587406ba029defe7dc2d876
I20260620 15:16:46.876667 5165 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a447a43e587406ba029defe7dc2d876 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:46 UTC) from {username='slave'} at 127.0.0.1:52594
I20260620 15:16:46.876741 5295 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a447a43e587406ba029defe7dc2d876 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:46 UTC) from {username='slave'} at 127.0.0.1:46930
I20260620 15:16:46.876773 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a447a43e587406ba029defe7dc2d876 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:16:46 UTC) from {username='slave'} at 127.0.0.1:41458
I20260620 15:16:46.878723 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 5103
I20260620 15:16:46.907238 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:46.880338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.0.97.65:34921)
0620 15:16:46.880581 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:46.880590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:46.903950 (+ 23360us) server_negotiation.cc:407] Connection header received
0620 15:16:46.904446 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:46.904455 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:46.904626 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:46.904718 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:46.906175 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:46.906184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:46.906191 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:46.906217 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:46.906264 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:46.906284 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:46.906417 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:46.906742 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 15:16:46.906819 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:16:46.908757 5225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:46.909942 5532 tablet_replica.cc:330] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: stopping tablet replica
I20260620 15:16:46.911222 5532 raft_consensus.cc:2229] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:16:46.912719 5532 raft_consensus.cc:2258] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:46.918244 5531 tablet_replica.cc:330] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: stopping tablet replica
I20260620 15:16:46.918242 5532 ts_tablet_manager.cc:1905] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:46.919945 5531 raft_consensus.cc:2229] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:46.920514 5531 raft_consensus.cc:2258] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:46.922274 5531 ts_tablet_manager.cc:1905] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:46.925350 5532 ts_tablet_manager.cc:1918] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:16:46.925508 5532 log.cc:1196] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/6a447a43e587406ba029defe7dc2d876
I20260620 15:16:46.926044 5532 ts_tablet_manager.cc:1939] T 6a447a43e587406ba029defe7dc2d876 P d3e1bd59bc244a5791068bd335e5cb65: Deleting consensus metadata
I20260620 15:16:46.927711 5031 catalog_manager.cc:4841] TS d3e1bd59bc244a5791068bd335e5cb65 (127.0.97.67:43865): tablet 6a447a43e587406ba029defe7dc2d876 (table test-table [id=346ee000f726478f9f0320a6f53324e4]) successfully deleted
I20260620 15:16:46.929638 5531 ts_tablet_manager.cc:1918] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:16:46.929893 5531 log.cc:1196] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/6a447a43e587406ba029defe7dc2d876
I20260620 15:16:46.930584 5531 ts_tablet_manager.cc:1939] T 6a447a43e587406ba029defe7dc2d876 P a1fdadbf15e54aaf81fa1045a3a6c38b: Deleting consensus metadata
I20260620 15:16:46.932533 5031 catalog_manager.cc:4841] TS a1fdadbf15e54aaf81fa1045a3a6c38b (127.0.97.66:38595): tablet 6a447a43e587406ba029defe7dc2d876 (table test-table [id=346ee000f726478f9f0320a6f53324e4]) successfully deleted
I20260620 15:16:47.009395 5530 tablet_replica.cc:330] T 6a447a43e587406ba029defe7dc2d876 P 9c57c9ff2c314f14b85993e52fc88570: stopping tablet replica
I20260620 15:16:47.009745 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 5103
W20260620 15:16:47.032253 5028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:34921 (error 108)
I20260620 15:16:47.032548 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 5233
I20260620 15:16:47.036118 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.033425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.0.97.66:38595)
0620 15:16:47.033643 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:47.033652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:47.033829 (+ 177us) server_negotiation.cc:407] Connection header received
0620 15:16:47.034057 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:47.034063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:47.034167 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:47.034279 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:47.035430 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:47.035437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:47.035441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:47.035460 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:47.035496 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:47.035512 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:47.035663 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:47.035769 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 15:16:47.035824 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:16:47.037053 5355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:47.038105 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.037625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34921 (local address 127.0.0.1:52620)
0620 15:16:47.037817 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:47.037952 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 15:16:47.038338 5028 catalog_manager.cc:4584] TS 9c57c9ff2c314f14b85993e52fc88570 (127.0.97.65:34921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a447a43e587406ba029defe7dc2d876: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34921: connect: Connection refused (error 111)
I20260620 15:16:47.091849 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.091316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34921 (local address 127.0.0.1:52632)
0620 15:16:47.091547 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:47.091664 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 15:16:47.092978 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.092608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34921 (local address 127.0.0.1:52646)
0620 15:16:47.092753 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:47.092831 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 15:16:47.124083 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 5233
I20260620 15:16:47.144263 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 5363
I20260620 15:16:47.147189 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.145101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41478 (local address 127.0.97.67:43865)
0620 15:16:47.145289 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:47.145295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:47.145314 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:16:47.145449 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:47.145455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:47.145510 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:47.145615 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:47.146544 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:47.146549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:47.146554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:47.146570 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:47.146612 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:47.146624 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:47.146775 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:47.146851 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 15:16:47.146912 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:16:47.148059 5485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:47.153180 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.152727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34921 (local address 127.0.0.1:52650)
0620 15:16:47.152918 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:47.153017 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 15:16:47.154111 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.153721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34921 (local address 127.0.0.1:52666)
0620 15:16:47.153874 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:47.153978 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:16:47.234757 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.234216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34921 (local address 127.0.0.1:52680)
0620 15:16:47.234436 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:47.234549 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:16:47.235845 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.235446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34921 (local address 127.0.0.1:52686)
0620 15:16:47.235638 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:47.235714 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 15:16:47.263552 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 5363
I20260620 15:16:47.282574 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 5013
I20260620 15:16:47.283685 5074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:47.386014 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 5013
2026-06-20T15:16:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2591 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 15:16:47.427343 389 mini_oidc.cc:152] Starting JWKS server
I20260620 15:16:47.429998 389 webserver.cc:460] Webserver started at https://127.0.0.1:37675/ using document root <none> and password file <none>
I20260620 15:16:47.430325 389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:16:47.431027 389 webserver.cc:460] Webserver started at http://127.0.0.1:43911/ using document root <none> and password file <none>
2026-06-20T15:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:47Z Disabled control of system clock
I20260620 15:16:47.456015 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46009
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:46009
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37675/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/testchainca.cert with env {}
W20260620 15:16:47.770223 5544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:47.770473 5544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:47.770557 5544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:47.777487 5544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:47.777575 5544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:47.777612 5544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:47.777670 5544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:47.777714 5544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:16:47.777765 5544 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:16:47.777813 5544 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37675/jwks.json
I20260620 15:16:47.787067 5544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:46009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37675/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:47.788152 5544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:47.789381 5544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:47.801214 5549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:47.801230 5552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:47.801234 5550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:47.802044 5544 server_base.cc:1034] running on GCE node
I20260620 15:16:47.802685 5544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:47.803856 5544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:47.805039 5544 hybrid_clock.cc:648] HybridClock initialized: now 1781968607804986 us; error 50 us; skew 500 ppm
I20260620 15:16:47.805364 5544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:16:47.810528 5544 webserver.cc:460] Webserver started at http://127.0.97.126:41281/ using document root <none> and password file <none>
I20260620 15:16:47.811188 5544 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:47.811300 5544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:47.811602 5544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:47.814332 5544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "0c1ff25686754f6fb81b706b732f04fd"
format_stamp: "Formatted at 2026-06-20 15:16:47 on dist-test-slave-7s64"
server_key: "fdaa439f4da27499c95d4f50ee69f6e7"
server_key_iv: "252f826559953291a1d0cd9412a703c9"
server_key_version: "encryptionkey@0"
I20260620 15:16:47.815105 5544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "0c1ff25686754f6fb81b706b732f04fd"
format_stamp: "Formatted at 2026-06-20 15:16:47 on dist-test-slave-7s64"
server_key: "fdaa439f4da27499c95d4f50ee69f6e7"
server_key_iv: "252f826559953291a1d0cd9412a703c9"
server_key_version: "encryptionkey@0"
I20260620 15:16:47.819904 5544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 15:16:47.823297 5558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:47.824553 5544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:47.824743 5544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "0c1ff25686754f6fb81b706b732f04fd"
format_stamp: "Formatted at 2026-06-20 15:16:47 on dist-test-slave-7s64"
server_key: "fdaa439f4da27499c95d4f50ee69f6e7"
server_key_iv: "252f826559953291a1d0cd9412a703c9"
server_key_version: "encryptionkey@0"
I20260620 15:16:47.824940 5544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:47.838271 5544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:47.838963 5544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:47.839219 5544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:47.856666 5544 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:46009
I20260620 15:16:47.856696 5609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:46009 every 8 connection(s)
I20260620 15:16:47.857859 5544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:47.860809 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 5544
I20260620 15:16:47.861012 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:47.861370 5610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:47.861670 389 external_mini_cluster.cc:1428] Setting key d78069b567885eb3e377657ac443dccd
I20260620 15:16:47.873303 5610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd: Bootstrap starting.
I20260620 15:16:47.874001 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.863327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.0.97.126:46009)
0620 15:16:47.864249 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:47.864259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:47.864296 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:16:47.865892 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:47.865912 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:47.866330 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:47.866694 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:47.867241 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:47.868456 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:47.869812 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:47.870450 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:47.870682 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:47.871853 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:47.871861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:47.871872 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:47.872361 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:47.872430 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:47.872448 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:47.872616 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:47.872839 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 15:16:47.873104 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":321,"threads_started":1}
I20260620 15:16:47.876652 5610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:47.877507 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:16:47.877625 5610 log.cc:824] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:47.880040 5610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd: No bootstrap required, opened a new log
I20260620 15:16:47.886643 5610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1ff25686754f6fb81b706b732f04fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46009 } }
I20260620 15:16:47.886966 5610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:47.887068 5610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1ff25686754f6fb81b706b732f04fd, State: Initialized, Role: FOLLOWER
I20260620 15:16:47.887559 5610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1ff25686754f6fb81b706b732f04fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46009 } }
I20260620 15:16:47.887775 5610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:47.887907 5610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:47.888078 5610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:47.890055 5610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1ff25686754f6fb81b706b732f04fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46009 } }
I20260620 15:16:47.890489 5610 leader_election.cc:302] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c1ff25686754f6fb81b706b732f04fd; no voters:
I20260620 15:16:47.891005 5610 leader_election.cc:288] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:47.891126 5615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:47.891767 5615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [term 1 LEADER]: Becoming Leader. State: Replica: 0c1ff25686754f6fb81b706b732f04fd, State: Running, Role: LEADER
I20260620 15:16:47.892258 5615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1ff25686754f6fb81b706b732f04fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46009 } }
I20260620 15:16:47.892908 5610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:47.895226 5617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c1ff25686754f6fb81b706b732f04fd. Latest consensus state: current_term: 1 leader_uuid: "0c1ff25686754f6fb81b706b732f04fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1ff25686754f6fb81b706b732f04fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46009 } } }
I20260620 15:16:47.895622 5617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:47.895431 5616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c1ff25686754f6fb81b706b732f04fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c1ff25686754f6fb81b706b732f04fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46009 } } }
I20260620 15:16:47.895877 5616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:47.905786 5620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:47.911605 5620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:47.926749 5620 catalog_manager.cc:1347] Generated new cluster ID: 42f20f9eea8645299a228c8b3d06bae3
I20260620 15:16:47.926867 5620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:47.948402 5620 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:47.950151 5620 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:47.960088 5620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c1ff25686754f6fb81b706b732f04fd: Generated new TSK 0
I20260620 15:16:47.961050 5620 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 15:16:48.012766 5640 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 15:16:48.012930 5640 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:16:48.013716 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:47.984964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51214 (local address 127.0.97.126:46009)
0620 15:16:47.985162 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:47.985171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:47.985740 (+ 569us) server_negotiation.cc:407] Connection header received
0620 15:16:47.986037 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:47.986040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:47.986101 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:47.986216 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:16:47.986860 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:47.987591 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:47.989721 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:47.990139 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:47.990226 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:47.990811 (+ 585us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:16:48.013238 (+ 22427us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 15:16:48.013493 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51214: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 15:16:48.013855 5613 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51214: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 15:16:48.015592 5639 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:16:47.984944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46009 (local address 127.0.0.1:51214)
0620 15:16:47.985628 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:47.985653 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:47.985827 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:47.986270 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:47.986275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:47.986288 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:16:47.986708 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:47.986720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:47.987721 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:47.987727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:47.989569 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:47.989585 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:47.990308 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:47.990314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:47.990506 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:47.990578 (+ 72us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:16:48.013500 (+ 22922us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 15:16:48.013830 (+ 330us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:46009: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":328,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.126:46009: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37675/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 15:16:48.017792 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 5544
I20260620 15:16:48.018857 5605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:48.109078 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 5544
2026-06-20T15:16:48Z chronyd exiting
I20260620 15:16:48.159065 389 test_util.cc:175] -----------------------------------------------
I20260620 15:16:48.159179 389 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781968566423581-389-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (736 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 15:16:48.162760 389 mini_oidc.cc:152] Starting JWKS server
I20260620 15:16:48.164404 389 webserver.cc:460] Webserver started at https://127.0.0.1:38913/ using document root <none> and password file <none>
I20260620 15:16:48.164678 389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:16:48.165302 389 webserver.cc:460] Webserver started at http://127.0.0.1:45993/ using document root <none> and password file <none>
2026-06-20T15:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:48Z Disabled control of system clock
I20260620 15:16:48.188009 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44121
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:34781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44121
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38913/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/testchainca.cert with env {}
W20260620 15:16:48.488036 5647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:48.488293 5647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:48.488374 5647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:48.495137 5647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:48.495222 5647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:48.495260 5647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:48.495306 5647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:48.495349 5647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:16:48.495406 5647 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:16:48.495446 5647 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38913/jwks.json
I20260620 15:16:48.504915 5647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44121
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38913/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:48.506003 5647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:48.507128 5647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:48.518000 5652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:48.518241 5655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:48.518684 5653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:48.519374 5647 server_base.cc:1034] running on GCE node
I20260620 15:16:48.520222 5647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:48.521742 5647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:48.522958 5647 hybrid_clock.cc:648] HybridClock initialized: now 1781968608522914 us; error 63 us; skew 500 ppm
I20260620 15:16:48.523270 5647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:16:48.525465 5647 webserver.cc:460] Webserver started at http://127.0.97.126:35233/ using document root <none> and password file <none>
I20260620 15:16:48.526109 5647 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:48.526237 5647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:48.526542 5647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:48.529259 5647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "ed2f1008b67e43afba571dab3d5adc9c"
format_stamp: "Formatted at 2026-06-20 15:16:48 on dist-test-slave-7s64"
server_key: "acfc0ff453ca51a8c48aeb843e5cdeb1"
server_key_iv: "270bfab5e05a37ff70b9754d46b3a0d5"
server_key_version: "encryptionkey@0"
I20260620 15:16:48.530015 5647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "ed2f1008b67e43afba571dab3d5adc9c"
format_stamp: "Formatted at 2026-06-20 15:16:48 on dist-test-slave-7s64"
server_key: "acfc0ff453ca51a8c48aeb843e5cdeb1"
server_key_iv: "270bfab5e05a37ff70b9754d46b3a0d5"
server_key_version: "encryptionkey@0"
I20260620 15:16:48.534795 5647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260620 15:16:48.538417 5661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:48.539637 5647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:16:48.539824 5647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "ed2f1008b67e43afba571dab3d5adc9c"
format_stamp: "Formatted at 2026-06-20 15:16:48 on dist-test-slave-7s64"
server_key: "acfc0ff453ca51a8c48aeb843e5cdeb1"
server_key_iv: "270bfab5e05a37ff70b9754d46b3a0d5"
server_key_version: "encryptionkey@0"
I20260620 15:16:48.540056 5647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:48.556529 5647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:48.557226 5647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:48.557484 5647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:48.575280 5647 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44121
I20260620 15:16:48.575313 5712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44121 every 8 connection(s)
I20260620 15:16:48.576571 5647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:48.580197 5713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:48.582429 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 5647
I20260620 15:16:48.582645 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:48.583406 389 external_mini_cluster.cc:1428] Setting key 86d625de79e07b82eea0c1ae1476f49b
I20260620 15:16:48.591821 5713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c: Bootstrap starting.
I20260620 15:16:48.594700 5713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:48.595777 5713 log.cc:824] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:48.596699 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:48.585340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.0.97.126:44121)
0620 15:16:48.586505 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:48.586515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:48.586567 (+ 52us) server_negotiation.cc:407] Connection header received
0620 15:16:48.588163 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:48.588184 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:48.588733 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:48.589116 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:48.589674 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:48.590921 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:48.592231 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:48.592937 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:48.593139 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:48.594234 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:48.594240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:48.594254 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:48.594762 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:48.594891 (+ 129us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:48.594922 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:48.595046 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:48.595262 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 15:16:48.595536 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":354,"threads_started":1}
I20260620 15:16:48.598470 5713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c: No bootstrap required, opened a new log
I20260620 15:16:48.600112 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:16:48.605712 5713 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f1008b67e43afba571dab3d5adc9c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44121 } }
I20260620 15:16:48.606101 5713 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:48.606202 5713 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2f1008b67e43afba571dab3d5adc9c, State: Initialized, Role: FOLLOWER
I20260620 15:16:48.606740 5713 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f1008b67e43afba571dab3d5adc9c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44121 } }
I20260620 15:16:48.606983 5713 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:48.607120 5713 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:48.607287 5713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:48.609303 5713 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f1008b67e43afba571dab3d5adc9c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44121 } }
I20260620 15:16:48.609781 5713 leader_election.cc:302] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed2f1008b67e43afba571dab3d5adc9c; no voters:
I20260620 15:16:48.610379 5713 leader_election.cc:288] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:48.610479 5718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:48.611137 5718 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [term 1 LEADER]: Becoming Leader. State: Replica: ed2f1008b67e43afba571dab3d5adc9c, State: Running, Role: LEADER
I20260620 15:16:48.611676 5718 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f1008b67e43afba571dab3d5adc9c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44121 } }
I20260620 15:16:48.612394 5713 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:48.614298 5719 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed2f1008b67e43afba571dab3d5adc9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f1008b67e43afba571dab3d5adc9c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44121 } } }
I20260620 15:16:48.614665 5719 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:48.614507 5720 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed2f1008b67e43afba571dab3d5adc9c. Latest consensus state: current_term: 1 leader_uuid: "ed2f1008b67e43afba571dab3d5adc9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed2f1008b67e43afba571dab3d5adc9c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44121 } } }
I20260620 15:16:48.614851 5720 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:48.617125 5725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:48.622922 5725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:48.630335 5725 catalog_manager.cc:1347] Generated new cluster ID: 309a0928bbe94b67b3e9b1151870733f
I20260620 15:16:48.630442 5725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:48.638512 5725 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:48.640161 5725 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:48.648631 5725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed2f1008b67e43afba571dab3d5adc9c: Generated new TSK 0
I20260620 15:16:48.649444 5725 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 15:16:48.688287 5743 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 15:16:48.688413 5743 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:16:48.689224 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:48.662323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.0.97.126:44121)
0620 15:16:48.662557 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:48.662565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:48.662930 (+ 365us) server_negotiation.cc:407] Connection header received
0620 15:16:48.663165 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:48.663169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:48.663235 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:48.663367 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:16:48.663891 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:48.664697 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:48.666780 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:48.667233 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:48.667350 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:48.667893 (+ 543us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:16:48.688733 (+ 20840us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 15:16:48.688991 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 15:16:48.689356 5716 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35288: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 15:16:48.689556 5742 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:16:48.662221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44121 (local address 127.0.0.1:35288)
0620 15:16:48.662826 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:48.662845 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:48.662984 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:48.663409 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:48.663414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:48.663426 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:16:48.663741 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:48.663754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:48.664877 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:48.664883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:48.666625 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:48.666640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:48.667391 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:48.667398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:48.667577 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:48.667619 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:16:48.688923 (+ 21304us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 15:16:48.689191 (+ 268us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:44121: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38913/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":326,"threads_started":1}
I20260620 15:16:48.690894 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 5647
I20260620 15:16:48.691920 5708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:48.778728 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 5647
2026-06-20T15:16:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (671 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 15:16:48.833853 389 mini_oidc.cc:152] Starting JWKS server
I20260620 15:16:48.835600 389 webserver.cc:460] Webserver started at https://127.0.0.1:35865/ using document root <none> and password file <none>
I20260620 15:16:48.835778 389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:16:48.836301 389 webserver.cc:460] Webserver started at http://127.0.0.1:34037/ using document root <none> and password file <none>
2026-06-20T15:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:48Z Disabled control of system clock
I20260620 15:16:48.859802 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36945
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36945
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35865/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 15:16:49.174235 5750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:49.174501 5750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:49.174649 5750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:49.181507 5750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:49.181598 5750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:49.181696 5750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:49.181799 5750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:49.181881 5750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:16:49.181969 5750 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:16:49.182024 5750 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35865/jwks.json
I20260620 15:16:49.190554 5750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35865/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:49.191738 5750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:49.193480 5750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:49.204741 5756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:49.204803 5758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:49.205010 5755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:49.205591 5750 server_base.cc:1034] running on GCE node
I20260620 15:16:49.206317 5750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:49.207551 5750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:49.208735 5750 hybrid_clock.cc:648] HybridClock initialized: now 1781968609208700 us; error 33 us; skew 500 ppm
I20260620 15:16:49.209074 5750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:16:49.211179 5750 webserver.cc:460] Webserver started at http://127.0.97.126:46561/ using document root <none> and password file <none>
I20260620 15:16:49.211797 5750 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:49.211884 5750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:49.212225 5750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:49.214921 5750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "16c25a513a1749fba0da24032f6204aa"
format_stamp: "Formatted at 2026-06-20 15:16:49 on dist-test-slave-7s64"
server_key: "c72fe0d8d852a4c5ca6ca374cb7589b9"
server_key_iv: "f8e14af658ea03f32e5439bafa96743c"
server_key_version: "encryptionkey@0"
I20260620 15:16:49.215654 5750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "16c25a513a1749fba0da24032f6204aa"
format_stamp: "Formatted at 2026-06-20 15:16:49 on dist-test-slave-7s64"
server_key: "c72fe0d8d852a4c5ca6ca374cb7589b9"
server_key_iv: "f8e14af658ea03f32e5439bafa96743c"
server_key_version: "encryptionkey@0"
I20260620 15:16:49.220567 5750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:16:49.223814 5764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:49.225164 5750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:49.225338 5750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "16c25a513a1749fba0da24032f6204aa"
format_stamp: "Formatted at 2026-06-20 15:16:49 on dist-test-slave-7s64"
server_key: "c72fe0d8d852a4c5ca6ca374cb7589b9"
server_key_iv: "f8e14af658ea03f32e5439bafa96743c"
server_key_version: "encryptionkey@0"
I20260620 15:16:49.225531 5750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:49.240332 5750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:49.241074 5750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:49.241344 5750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:49.258893 5750 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36945
I20260620 15:16:49.258916 5815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36945 every 8 connection(s)
I20260620 15:16:49.260200 5750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:49.263721 5816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:49.264961 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 5750
I20260620 15:16:49.265161 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:49.265951 389 external_mini_cluster.cc:1428] Setting key ed05caf2f2788eefe046895ee15fa393
I20260620 15:16:49.277179 5816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa: Bootstrap starting.
I20260620 15:16:49.279136 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:49.267796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.0.97.126:36945)
0620 15:16:49.268885 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:49.268900 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:49.268944 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:16:49.270521 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:49.270542 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:49.270974 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:49.271281 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:49.271795 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:49.273108 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:49.274917 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:49.275592 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:49.275725 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:49.276979 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:49.276986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:49.276997 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:49.277571 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:49.277642 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:49.277662 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:49.277752 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:49.277969 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 15:16:49.278267 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":443,"threads_started":1}
I20260620 15:16:49.280184 5816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:49.281221 5816 log.cc:824] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:49.283156 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:16:49.284109 5816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa: No bootstrap required, opened a new log
I20260620 15:16:49.291033 5816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25a513a1749fba0da24032f6204aa" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36945 } }
I20260620 15:16:49.291435 5816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:49.291525 5816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c25a513a1749fba0da24032f6204aa, State: Initialized, Role: FOLLOWER
I20260620 15:16:49.292052 5816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25a513a1749fba0da24032f6204aa" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36945 } }
I20260620 15:16:49.292303 5816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:49.292457 5816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:49.292644 5816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:49.294572 5816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25a513a1749fba0da24032f6204aa" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36945 } }
I20260620 15:16:49.295025 5816 leader_election.cc:302] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16c25a513a1749fba0da24032f6204aa; no voters:
I20260620 15:16:49.295591 5816 leader_election.cc:288] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:49.295799 5821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:49.296957 5821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [term 1 LEADER]: Becoming Leader. State: Replica: 16c25a513a1749fba0da24032f6204aa, State: Running, Role: LEADER
I20260620 15:16:49.297489 5816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:49.297465 5821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25a513a1749fba0da24032f6204aa" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36945 } }
I20260620 15:16:49.300721 5823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16c25a513a1749fba0da24032f6204aa. Latest consensus state: current_term: 1 leader_uuid: "16c25a513a1749fba0da24032f6204aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25a513a1749fba0da24032f6204aa" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36945 } } }
I20260620 15:16:49.301177 5823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:49.300741 5822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16c25a513a1749fba0da24032f6204aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16c25a513a1749fba0da24032f6204aa" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36945 } } }
I20260620 15:16:49.302259 5822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:49.313760 5831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:49.318126 5831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:49.326177 5831 catalog_manager.cc:1347] Generated new cluster ID: 2855b8f9714d4413a3735bc5fe3f73c3
I20260620 15:16:49.326303 5831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:49.347224 5831 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:49.348641 5831 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:49.359741 5831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16c25a513a1749fba0da24032f6204aa: Generated new TSK 0
I20260620 15:16:49.360787 5831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:16:49.391695 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:49.383624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.0.97.126:36945)
0620 15:16:49.383799 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:49.383805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:49.384339 (+ 534us) server_negotiation.cc:407] Connection header received
0620 15:16:49.384542 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:49.384547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:49.384647 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:49.384792 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:16:49.385268 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:49.386376 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:49.387667 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:49.388341 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:49.388472 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:49.388949 (+ 477us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:16:49.391116 (+ 2167us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 15:16:49.391415 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 15:16:49.391809 5819 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52674: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 15:16:49.393477 5845 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:16:49.383667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36945 (local address 127.0.0.1:52674)
0620 15:16:49.384252 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:49.384267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:49.384375 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:49.384858 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:49.384862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:49.384872 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:16:49.385137 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:49.385145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:49.386506 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:49.386509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:49.387533 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:49.387543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:49.388567 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:49.388572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:49.388691 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:49.388721 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:16:49.391301 (+ 2580us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 15:16:49.393064 (+ 1763us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:36945: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":1488,"thread_start_us":334,"threads_started":1}
I20260620 15:16:49.395336 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 5750
I20260620 15:16:49.396324 5811 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 15:16:49.403381 5846 webserver.cc:533] Webserver: sslize failed:
E20260620 15:16:49.403487 5846 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:16:49.486135 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 5750
2026-06-20T15:16:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (706 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T15:16:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:49Z Disabled control of system clock
I20260620 15:16:49.564652 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46705
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:46705
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/testchainca.cert with env {}
W20260620 15:16:49.870258 5851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:49.870530 5851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:49.870676 5851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:49.877600 5851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:49.877694 5851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:49.877784 5851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:49.877885 5851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:49.877962 5851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:16:49.878068 5851 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:16:49.878142 5851 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 15:16:49.887629 5851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:46705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:49.888885 5851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:49.890111 5851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:49.901044 5857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:49.901000 5859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:49.901077 5856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:49.901860 5851 server_base.cc:1034] running on GCE node
I20260620 15:16:49.902601 5851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:49.903816 5851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:49.905041 5851 hybrid_clock.cc:648] HybridClock initialized: now 1781968609905009 us; error 47 us; skew 500 ppm
I20260620 15:16:49.905357 5851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:16:49.907450 5851 webserver.cc:460] Webserver started at http://127.0.97.126:32791/ using document root <none> and password file <none>
I20260620 15:16:49.908077 5851 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:49.908207 5851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:49.908563 5851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:49.911190 5851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "e1ec336681354fd783133537d7e5c66d"
format_stamp: "Formatted at 2026-06-20 15:16:49 on dist-test-slave-7s64"
server_key: "0bc2ea0e86b3b2bdeecf061f577b0507"
server_key_iv: "2b618192405feab4b5100592d09dc701"
server_key_version: "encryptionkey@0"
I20260620 15:16:49.911912 5851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "e1ec336681354fd783133537d7e5c66d"
format_stamp: "Formatted at 2026-06-20 15:16:49 on dist-test-slave-7s64"
server_key: "0bc2ea0e86b3b2bdeecf061f577b0507"
server_key_iv: "2b618192405feab4b5100592d09dc701"
server_key_version: "encryptionkey@0"
I20260620 15:16:49.916455 5851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260620 15:16:49.919664 5865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:49.920949 5851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:49.921123 5851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "e1ec336681354fd783133537d7e5c66d"
format_stamp: "Formatted at 2026-06-20 15:16:49 on dist-test-slave-7s64"
server_key: "0bc2ea0e86b3b2bdeecf061f577b0507"
server_key_iv: "2b618192405feab4b5100592d09dc701"
server_key_version: "encryptionkey@0"
I20260620 15:16:49.921321 5851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:49.941056 5851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:49.941763 5851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:49.942023 5851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:49.959932 5851 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:46705
I20260620 15:16:49.959967 5916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:46705 every 8 connection(s)
I20260620 15:16:49.961177 5851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:49.964728 5917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:49.969846 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 5851
I20260620 15:16:49.970069 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:49.970975 389 external_mini_cluster.cc:1428] Setting key 21e8c024ac999897c4e52c357d512f2d
I20260620 15:16:49.976058 5917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d: Bootstrap starting.
I20260620 15:16:49.979162 5917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:49.980247 5917 log.cc:824] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:49.983175 5917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d: No bootstrap required, opened a new log
I20260620 15:16:49.983595 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:49.972878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.0.97.126:46705)
0620 15:16:49.973818 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:49.973827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:49.973864 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:16:49.975488 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:49.975509 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:49.975913 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:49.976304 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:49.976859 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:49.978083 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:49.979310 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:49.979973 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:49.980166 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:49.981389 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:49.981409 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:49.981428 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:16:49.981955 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:49.982025 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:16:49.982044 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:49.982127 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:49.982383 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 15:16:49.982694 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":327,"threads_started":1}
I20260620 15:16:49.987380 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:16:49.991170 5917 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ec336681354fd783133537d7e5c66d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46705 } }
I20260620 15:16:49.991508 5917 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:49.991621 5917 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ec336681354fd783133537d7e5c66d, State: Initialized, Role: FOLLOWER
I20260620 15:16:49.992228 5917 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ec336681354fd783133537d7e5c66d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46705 } }
I20260620 15:16:49.992511 5917 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:49.992676 5917 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:49.992849 5917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:49.994848 5917 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ec336681354fd783133537d7e5c66d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46705 } }
I20260620 15:16:49.995314 5917 leader_election.cc:302] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1ec336681354fd783133537d7e5c66d; no voters:
I20260620 15:16:49.995846 5917 leader_election.cc:288] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:49.996024 5922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:49.997021 5922 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [term 1 LEADER]: Becoming Leader. State: Replica: e1ec336681354fd783133537d7e5c66d, State: Running, Role: LEADER
I20260620 15:16:49.997486 5922 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ec336681354fd783133537d7e5c66d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46705 } }
I20260620 15:16:49.997864 5917 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:50.000334 5923 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1ec336681354fd783133537d7e5c66d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ec336681354fd783133537d7e5c66d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46705 } } }
I20260620 15:16:50.000311 5924 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1ec336681354fd783133537d7e5c66d. Latest consensus state: current_term: 1 leader_uuid: "e1ec336681354fd783133537d7e5c66d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1ec336681354fd783133537d7e5c66d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46705 } } }
I20260620 15:16:50.000736 5923 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:50.000808 5924 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:50.003796 5929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:50.009193 5929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:50.017859 5929 catalog_manager.cc:1347] Generated new cluster ID: 269f715b4667495e90789bc2327e9a1e
I20260620 15:16:50.017980 5929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:50.024211 5929 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:50.026031 5929 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:50.037221 5929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1ec336681354fd783133537d7e5c66d: Generated new TSK 0
I20260620 15:16:50.038066 5929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:16:50.124049 5920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:50.050257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.0.97.126:46705)
0620 15:16:50.050436 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:50.050442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:50.050776 (+ 334us) server_negotiation.cc:407] Connection header received
0620 15:16:50.050975 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:50.050980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:50.051075 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:50.051223 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:16:50.051629 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:50.052760 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:50.054138 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:50.054838 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:50.054950 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:50.055348 (+ 398us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:16:50.123321 (+ 67973us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 15:16:50.123732 (+ 411us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36288: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 15:16:50.124157 5946 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:16:50.050118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46705 (local address 127.0.0.1:36288)
0620 15:16:50.050681 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:50.050696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:50.050807 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:50.051233 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:50.051236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:50.051245 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:16:50.051488 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:50.051495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:50.052893 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:50.052897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:50.054002 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:50.054013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:50.054986 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:50.054990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:50.055106 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:50.055134 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:16:50.123616 (+ 68482us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 15:16:50.123848 (+ 232us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:46705: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":301,"threads_started":1}
W20260620 15:16:50.124195 5920 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36288: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260620 15:16:50.125425 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 5851
I20260620 15:16:50.126648 5912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:50.221571 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 5851
2026-06-20T15:16:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (714 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (390 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (390 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:51 dist-test-slave-7s64 krb5kdc[5961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:51 dist-test-slave-7s64 krb5kdc[5961](info): set up 2 sockets
Jun 20 15:16:51 dist-test-slave-7s64 krb5kdc[5961](info): commencing operation
krb5kdc: starting...
W20260620 15:16:52.081128 389 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:52 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:16:52.274219 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43117
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:52.583659 5974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:52.583971 5974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:52.584118 5974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:52.591301 5974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:52.591445 5974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:52.591549 5974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:52.591652 5974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:52.591720 5974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:52.601507 5974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:52.602809 5974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:52.604102 5974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:52.615296 5979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:52.615280 5980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:52.615309 5982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:52.616456 5974 server_base.cc:1034] running on GCE node
I20260620 15:16:52.617198 5974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:52.618433 5974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:52.619637 5974 hybrid_clock.cc:648] HybridClock initialized: now 1781968612619600 us; error 31 us; skew 500 ppm
I20260620 15:16:52.620003 5974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:52 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:52.622941 5974 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:16:52.624249 5974 webserver.cc:460] Webserver started at http://127.0.97.126:38533/ using document root <none> and password file <none>
I20260620 15:16:52.624928 5974 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:52.625065 5974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:52.625365 5974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:52.628064 5974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "92a4c115e9db41f4a924c10a7c6f437f"
format_stamp: "Formatted at 2026-06-20 15:16:52 on dist-test-slave-7s64"
server_key: "dc7c6c61fa46cad9394372b86266caa0"
server_key_iv: "a9fb3b0fd64fb8c50214ccb7a0895edf"
server_key_version: "encryptionkey@0"
I20260620 15:16:52.628835 5974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "92a4c115e9db41f4a924c10a7c6f437f"
format_stamp: "Formatted at 2026-06-20 15:16:52 on dist-test-slave-7s64"
server_key: "dc7c6c61fa46cad9394372b86266caa0"
server_key_iv: "a9fb3b0fd64fb8c50214ccb7a0895edf"
server_key_version: "encryptionkey@0"
I20260620 15:16:52.633450 5974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:16:52.636654 5989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:52.637884 5974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:16:52.638046 5974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "92a4c115e9db41f4a924c10a7c6f437f"
format_stamp: "Formatted at 2026-06-20 15:16:52 on dist-test-slave-7s64"
server_key: "dc7c6c61fa46cad9394372b86266caa0"
server_key_iv: "a9fb3b0fd64fb8c50214ccb7a0895edf"
server_key_version: "encryptionkey@0"
I20260620 15:16:52.638221 5974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:52.661031 5974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:52.665041 5974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:52.665298 5974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:52.682309 5974 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43117
I20260620 15:16:52.682366 6040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43117 every 8 connection(s)
I20260620 15:16:52.683496 5974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:52.686908 6041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:52.690192 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 5974
I20260620 15:16:52.690425 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:52.691020 389 external_mini_cluster.cc:1428] Setting key f656464bd06ce0f313695892484ce08a
I20260620 15:16:52.698489 6041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f: Bootstrap starting.
I20260620 15:16:52.702016 6041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:52.703243 6041 log.cc:824] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:52.705904 6041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f: No bootstrap required, opened a new log
Jun 20 15:16:52 dist-test-slave-7s64 krb5kdc[5961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:52.714627 6044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:52.693027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.0.97.126:43117)
0620 15:16:52.693793 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:52.693807 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:52.693868 (+ 61us) server_negotiation.cc:407] Connection header received
0620 15:16:52.696874 (+ 3006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:52.696901 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:52.697355 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:52.697752 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:52.699001 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:52.701007 (+ 2006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:52.702287 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:52.703316 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:52.703483 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:52.707790 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:52.707797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:52.707813 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:52.707843 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:52.710896 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:52.711650 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:52.711656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:52.711662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:52.711808 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:52.712196 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:52.712199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:52.712201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:52.712711 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:52.712923 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:52.713357 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 15:16:52.713662 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":321,"threads_started":1}
I20260620 15:16:52.715090 6041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a4c115e9db41f4a924c10a7c6f437f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43117 } }
I20260620 15:16:52.715446 6041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:52.715569 6041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a4c115e9db41f4a924c10a7c6f437f, State: Initialized, Role: FOLLOWER
I20260620 15:16:52.716158 6041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a4c115e9db41f4a924c10a7c6f437f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43117 } }
I20260620 15:16:52.716396 6041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:52.716579 6041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:52.716750 6041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:52.719106 6041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a4c115e9db41f4a924c10a7c6f437f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43117 } }
I20260620 15:16:52.719600 6041 leader_election.cc:302] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92a4c115e9db41f4a924c10a7c6f437f; no voters:
I20260620 15:16:52.720155 6041 leader_election.cc:288] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:52.720291 6046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:52.720974 6046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [term 1 LEADER]: Becoming Leader. State: Replica: 92a4c115e9db41f4a924c10a7c6f437f, State: Running, Role: LEADER
I20260620 15:16:52.721459 6046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a4c115e9db41f4a924c10a7c6f437f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43117 } }
I20260620 15:16:52.722100 6041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:52.724296 6047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92a4c115e9db41f4a924c10a7c6f437f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a4c115e9db41f4a924c10a7c6f437f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43117 } } }
I20260620 15:16:52.724776 6047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:52.724431 6048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92a4c115e9db41f4a924c10a7c6f437f. Latest consensus state: current_term: 1 leader_uuid: "92a4c115e9db41f4a924c10a7c6f437f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a4c115e9db41f4a924c10a7c6f437f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43117 } } }
I20260620 15:16:52.725272 6048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:52.729316 6055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:52.733472 6055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:52.741677 6055 catalog_manager.cc:1347] Generated new cluster ID: 87aa1435f50846499c08d24f6ae516cb
I20260620 15:16:52.741781 6055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:52.769361 6055 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:52.770716 6055 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:52.785755 6055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92a4c115e9db41f4a924c10a7c6f437f: Generated new TSK 0
I20260620 15:16:52.786612 6055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:16:52.831756 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43117
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:53.150017 6067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:53.150254 6067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:53.150440 6067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:53.157505 6067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:53.157595 6067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:53.157799 6067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:53.167023 6067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:53.168231 6067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:53.169512 6067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:53.182106 6073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:53.182312 6072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:53.182281 6075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:53.183029 6067 server_base.cc:1034] running on GCE node
I20260620 15:16:53.183588 6067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:53.184415 6067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:53.185674 6067 hybrid_clock.cc:648] HybridClock initialized: now 1781968613185633 us; error 37 us; skew 500 ppm
I20260620 15:16:53.185976 6067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:53 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:53.188934 6067 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:16:53.190227 6067 webserver.cc:460] Webserver started at http://127.0.97.65:43947/ using document root <none> and password file <none>
I20260620 15:16:53.190857 6067 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:53.190999 6067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:53.191279 6067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:53.194193 6067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "04c6fef067174ca3ad1207bc092f2be1"
format_stamp: "Formatted at 2026-06-20 15:16:53 on dist-test-slave-7s64"
server_key: "e1e8c8c108adc4b422ffbbf19967a721"
server_key_iv: "71d89f947fd1ee37d2c2c62616cd21c8"
server_key_version: "encryptionkey@0"
I20260620 15:16:53.194932 6067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "04c6fef067174ca3ad1207bc092f2be1"
format_stamp: "Formatted at 2026-06-20 15:16:53 on dist-test-slave-7s64"
server_key: "e1e8c8c108adc4b422ffbbf19967a721"
server_key_iv: "71d89f947fd1ee37d2c2c62616cd21c8"
server_key_version: "encryptionkey@0"
I20260620 15:16:53.199755 6067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:16:53.202934 6082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:53.204195 6067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:16:53.204358 6067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "04c6fef067174ca3ad1207bc092f2be1"
format_stamp: "Formatted at 2026-06-20 15:16:53 on dist-test-slave-7s64"
server_key: "e1e8c8c108adc4b422ffbbf19967a721"
server_key_iv: "71d89f947fd1ee37d2c2c62616cd21c8"
server_key_version: "encryptionkey@0"
I20260620 15:16:53.204593 6067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:53.265998 6067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:53.270166 6067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:53.270443 6067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:53.271365 6067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:53.272660 6067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:53.272758 6067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:53.272909 6067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:53.272989 6067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:53.305476 6067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:44525
I20260620 15:16:53.305557 6194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:44525 every 8 connection(s)
I20260620 15:16:53.306658 6067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:53.309965 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 6067
I20260620 15:16:53.310194 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:53.311096 389 external_mini_cluster.cc:1428] Setting key cbc2e2eb2287ee9e08d591dbb34d8d0b
Jun 20 15:16:53 dist-test-slave-7s64 krb5kdc[5961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:53.328357 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:53.309488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:36657 (local address 127.0.97.126:43117)
0620 15:16:53.309903 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:53.309907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:53.311070 (+ 1163us) server_negotiation.cc:407] Connection header received
0620 15:16:53.313066 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:53.313071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:53.313141 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:53.313282 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:53.314628 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:53.315843 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:53.317397 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:53.318146 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:53.318321 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:53.322126 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:53.322132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:53.322135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:53.322166 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:53.325011 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:53.325829 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:53.325840 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:53.325860 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:53.325995 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:53.326447 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:53.326452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:53.326456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:53.326851 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:53.327007 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:53.327705 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 15:16:53.327940 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":225,"threads_started":1}
I20260620 15:16:53.329241 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:53.310146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43117 (local address 127.0.97.65:36657)
0620 15:16:53.310918 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:53.310958 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:53.312787 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:53.313477 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:53.313484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:53.313930 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:53.314444 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:53.314470 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:53.316007 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:53.316010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:53.317254 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:53.317265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:53.318346 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:53.318350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:53.318530 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:53.319257 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:53.319296 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:53.321882 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:53.325183 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:53.325192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:53.325207 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:53.325695 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:53.326107 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:53.326114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:53.326120 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:53.326322 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:53.326980 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:53.326988 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:53.327324 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 15:16:53.327592 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 15:16:53.328090 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":352,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:16:53.330857 6195 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43117
I20260620 15:16:53.331131 6195 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:53.331873 6195 heartbeater.cc:511] Master 127.0.97.126:43117 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260620 15:16:53.334285 6006 ts_manager.cc:194] Registered new tserver with Master: 04c6fef067174ca3ad1207bc092f2be1 (127.0.97.65:44525)
I20260620 15:16:53.336654 6006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:36657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:16:53.356016 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43117
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:53.671093 6201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:53.671370 6201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:53.671557 6201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:53.678362 6201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:53.678491 6201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:53.678692 6201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:53.688321 6201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:53.689419 6201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:53.690614 6201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:53.703002 6206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:53.703083 6207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:53.703107 6209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:53.703711 6201 server_base.cc:1034] running on GCE node
I20260620 15:16:53.704241 6201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:53.705147 6201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:53.706367 6201 hybrid_clock.cc:648] HybridClock initialized: now 1781968613706332 us; error 50 us; skew 500 ppm
I20260620 15:16:53.706667 6201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:53 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:53.709682 6201 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:16:53.711043 6201 webserver.cc:460] Webserver started at http://127.0.97.66:45119/ using document root <none> and password file <none>
I20260620 15:16:53.711658 6201 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:53.711795 6201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:53.712074 6201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:53.714730 6201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "384e6815f919424d8aed92421a742200"
format_stamp: "Formatted at 2026-06-20 15:16:53 on dist-test-slave-7s64"
server_key: "33f1fd9372a204620d5ae5b96d75a6dc"
server_key_iv: "2578df2315ec187813257ca09f048c0e"
server_key_version: "encryptionkey@0"
I20260620 15:16:53.715454 6201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "384e6815f919424d8aed92421a742200"
format_stamp: "Formatted at 2026-06-20 15:16:53 on dist-test-slave-7s64"
server_key: "33f1fd9372a204620d5ae5b96d75a6dc"
server_key_iv: "2578df2315ec187813257ca09f048c0e"
server_key_version: "encryptionkey@0"
I20260620 15:16:53.720201 6201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260620 15:16:53.723856 6216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:53.725364 6201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:53.725523 6201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "384e6815f919424d8aed92421a742200"
format_stamp: "Formatted at 2026-06-20 15:16:53 on dist-test-slave-7s64"
server_key: "33f1fd9372a204620d5ae5b96d75a6dc"
server_key_iv: "2578df2315ec187813257ca09f048c0e"
server_key_version: "encryptionkey@0"
I20260620 15:16:53.725708 6201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:53.738019 6201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:53.741871 6201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:53.742158 6201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:53.743031 6201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:53.744263 6201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:53.744364 6201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:53.744535 6201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:53.744614 6201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 15:16:53.776737 6201 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37215
I20260620 15:16:53.776783 6328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37215 every 8 connection(s)
I20260620 15:16:53.777925 6201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:53.782356 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 6201
I20260620 15:16:53.782526 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:53.782936 389 external_mini_cluster.cc:1428] Setting key 19dbd7b958882e482770cf93475f8cf6
Jun 20 15:16:53 dist-test-slave-7s64 krb5kdc[5961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:53.799862 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:53.780282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:45207 (local address 127.0.97.126:43117)
0620 15:16:53.780507 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:53.780514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:53.781671 (+ 1157us) server_negotiation.cc:407] Connection header received
0620 15:16:53.783784 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:53.783790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:53.783866 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:53.783973 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:53.785731 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:53.786945 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:53.788450 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:53.789187 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:53.789318 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:53.793056 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:53.793065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:53.793073 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:53.793117 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:53.796685 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:53.797351 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:53.797359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:53.797366 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:53.797520 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:53.798006 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:53.798013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:53.798020 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:53.798385 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:53.798543 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:53.799198 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 15:16:53.799438 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:16:53.800606 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:53.780767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43117 (local address 127.0.97.66:45207)
0620 15:16:53.781505 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:53.781544 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:53.783470 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:53.784270 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:53.784281 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:53.785009 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:53.785565 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:53.785581 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:53.787092 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:53.787096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:53.788274 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:53.788286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:53.789369 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:53.789376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:53.789553 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:53.790232 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:53.790254 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:53.792835 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:53.796814 (+ 3979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:53.796820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:53.796831 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:53.797218 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:53.797666 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:53.797670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:53.797672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:53.797871 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:53.798541 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:53.798547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:53.798818 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 15:16:53.799046 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 15:16:53.799481 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":303,"threads_started":1}
I20260620 15:16:53.801954 6329 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43117
I20260620 15:16:53.802234 6329 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:53.802902 6329 heartbeater.cc:511] Master 127.0.97.126:43117 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260620 15:16:53.804296 6006 ts_manager.cc:194] Registered new tserver with Master: 384e6815f919424d8aed92421a742200 (127.0.97.66:37215)
I20260620 15:16:53.805181 6006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:45207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:16:53.829408 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43117
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:54.135614 6334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:54.135903 6334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:54.136070 6334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:54.143186 6334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:54.143316 6334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:54.143483 6334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:54.152601 6334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:54.153770 6334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:54.155020 6334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:54.167572 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:54.167623 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:54.167778 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:54.168881 6334 server_base.cc:1034] running on GCE node
I20260620 15:16:54.169648 6334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:54.170555 6334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:54.171753 6334 hybrid_clock.cc:648] HybridClock initialized: now 1781968614171690 us; error 55 us; skew 500 ppm
I20260620 15:16:54.172019 6334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:54.174978 6334 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:16:54.176358 6334 webserver.cc:460] Webserver started at http://127.0.97.67:33795/ using document root <none> and password file <none>
I20260620 15:16:54.177042 6334 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:54.177184 6334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:54.177466 6334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:54.180150 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "379739e308fe4fc7af254612c9ea3d9a"
format_stamp: "Formatted at 2026-06-20 15:16:54 on dist-test-slave-7s64"
server_key: "2489b9c096aeca074c956b574e00ec8a"
server_key_iv: "3984b53da6dc8a8edeead43ac1ab112d"
server_key_version: "encryptionkey@0"
I20260620 15:16:54.180917 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "379739e308fe4fc7af254612c9ea3d9a"
format_stamp: "Formatted at 2026-06-20 15:16:54 on dist-test-slave-7s64"
server_key: "2489b9c096aeca074c956b574e00ec8a"
server_key_iv: "3984b53da6dc8a8edeead43ac1ab112d"
server_key_version: "encryptionkey@0"
I20260620 15:16:54.185545 6334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:16:54.188717 6349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:54.189865 6334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:16:54.190026 6334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "379739e308fe4fc7af254612c9ea3d9a"
format_stamp: "Formatted at 2026-06-20 15:16:54 on dist-test-slave-7s64"
server_key: "2489b9c096aeca074c956b574e00ec8a"
server_key_iv: "3984b53da6dc8a8edeead43ac1ab112d"
server_key_version: "encryptionkey@0"
I20260620 15:16:54.190207 6334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:54.220629 6334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:54.224567 6334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:54.224826 6334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:54.225764 6334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:54.226998 6334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:54.227097 6334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:54.227238 6334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:54.227309 6334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:54.259948 6334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:43427
I20260620 15:16:54.260038 6461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:43427 every 8 connection(s)
I20260620 15:16:54.261191 6334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:54.265174 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 6334
I20260620 15:16:54.265393 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:54.265955 389 external_mini_cluster.cc:1428] Setting key 0ea393eabc84e02d66bf417d642ac6a0
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:16:54.281565 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:54.263688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:33555 (local address 127.0.97.126:43117)
0620 15:16:54.263874 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:54.263878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:54.265166 (+ 1288us) server_negotiation.cc:407] Connection header received
0620 15:16:54.267597 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:54.267601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:54.267663 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:54.267734 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:54.269539 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.270256 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.271724 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.272097 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.272182 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:54.275632 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:54.275636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:54.275640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:54.275668 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:54.278292 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.278958 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.278962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.278964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.279082 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.279611 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.279616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.279620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.280009 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:54.280177 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:54.281116 (+ 939us) server_negotiation.cc:299] Negotiation successful
0620 15:16:54.281327 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:16:54.282402 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:54.264129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43117 (local address 127.0.97.67:33555)
0620 15:16:54.264963 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:54.265023 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:54.267295 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:54.268026 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:54.268038 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:54.268646 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:54.269380 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:54.269398 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.270415 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.270419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:54.271603 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:54.271615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.272265 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.272269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:54.272449 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:54.273160 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:54.273185 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:54.275451 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:54.278443 (+ 2992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:54.278449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:54.278459 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:54.278838 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:54.279213 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:54.279218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:54.279221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:54.279468 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:54.280208 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:54.280216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:54.280634 (+ 418us) client_negotiation.cc:769] Sending connection context
0620 15:16:54.280912 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 15:16:54.281401 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":348,"threads_started":1}
I20260620 15:16:54.283550 6462 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43117
I20260620 15:16:54.283814 6462 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:54.284452 6462 heartbeater.cc:511] Master 127.0.97.126:43117 requested a full tablet report, sending...
I20260620 15:16:54.285710 6006 ts_manager.cc:194] Registered new tserver with Master: 379739e308fe4fc7af254612c9ea3d9a (127.0.97.67:43427)
I20260620 15:16:54.286429 6006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:33555
I20260620 15:16:54.293149 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/test-admin.keytab.
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:54.318068 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 15:16:54.322964 6469 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:16:54.320426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.0.97.65:44525)
0620 15:16:54.321142 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:54.321157 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:54.321179 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:16:54.321233 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:54.321238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:54.321408 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:54.321569 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:54.322469 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60680: BlockingRecv error: recv got EOF from 127.0.0.1:60680 (error 108)
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":334,"threads_started":1}
W20260620 15:16:54.323009 389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:54.324584 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:54.326025 389 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:54.339702 6195 heartbeater.cc:503] Master 127.0.97.126:43117 was elected leader, sending a full tablet report...
I20260620 15:16:54.340152 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:54.327488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.0.97.65:44525)
0620 15:16:54.327680 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:54.327686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:54.327894 (+ 208us) server_negotiation.cc:407] Connection header received
0620 15:16:54.328011 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:54.328014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:54.328063 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:54.328141 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:54.329208 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.330125 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.331222 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.331719 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.331811 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:54.335025 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:54.335034 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:54.335045 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:54.335076 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:54.337746 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.338269 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.338275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.338278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.338382 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.338689 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.338692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.338694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.338993 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:54.339178 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:54.339512 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 15:16:54.339784 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 15:16:54.345530 6469 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:16:54.343998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.0.97.65:44525)
0620 15:16:54.344208 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:54.344213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:54.344374 (+ 161us) server_negotiation.cc:407] Connection header received
0620 15:16:54.344474 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:54.344477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:54.344585 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:54.344659 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:54.345303 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60690: BlockingRecv error: recv got EOF from 127.0.0.1:60690 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260620 15:16:54.345691 389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:54.346995 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:54.348284 389 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:54.361331 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:54.349546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.0.97.65:44525)
0620 15:16:54.349675 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:54.349679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:54.350020 (+ 341us) server_negotiation.cc:407] Connection header received
0620 15:16:54.350111 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:54.350115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:54.350159 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:54.350230 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:54.351067 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.351708 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.352951 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.353358 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.353447 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:54.356571 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:54.356577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:54.356580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:54.356606 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:54.358973 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.359621 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.359627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.359631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.359749 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.360108 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.360114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.360117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.360355 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:54.360648 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:54.360909 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 15:16:54.361098 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 15:16:54.365149 6469 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:16:54.363669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.0.97.65:44525)
0620 15:16:54.363838 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:54.363845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:54.364029 (+ 184us) server_negotiation.cc:407] Connection header received
0620 15:16:54.364131 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:54.364134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:54.364190 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:54.364261 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:54.364958 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60702: BlockingRecv error: recv got EOF from 127.0.0.1:60702 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 15:16:54.365437 389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781968566423581-389-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:54.366740 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:54.367942 389 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:16:54.380682 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:54.369109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.0.97.65:44525)
0620 15:16:54.369292 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:54.369298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:54.369515 (+ 217us) server_negotiation.cc:407] Connection header received
0620 15:16:54.369673 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:54.369678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:54.369734 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:54.369838 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:54.370778 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.371450 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.372662 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.373064 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.373192 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:54.376153 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:54.376157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:54.376160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:54.376186 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:54.378557 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.379117 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.379120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.379122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.379236 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.379523 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.379526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.379528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.379797 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:54.380012 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:54.380233 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 15:16:54.380371 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:16:54.382040 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 6067
I20260620 15:16:54.392146 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:54.382644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.0.97.65:44525)
0620 15:16:54.382824 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:54.382829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:54.383016 (+ 187us) server_negotiation.cc:407] Connection header received
0620 15:16:54.383119 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:54.383122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:54.383167 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:54.383230 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:54.383998 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.384661 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.385754 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.386130 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.386213 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:54.387979 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:54.387983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:54.387986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:54.388007 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:54.390130 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.390660 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.390663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.390665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.390791 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.391076 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.391079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.391081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.391275 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:54.391442 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:54.391675 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 15:16:54.391812 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:16:54.392709 6190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:54.474460 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 6067
I20260620 15:16:54.490577 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 6201
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:16:54.505053 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:54.491325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.0.97.66:37215)
0620 15:16:54.491922 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:54.491929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:54.491954 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:16:54.492013 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:54.492019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:54.492237 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:54.492364 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:54.493436 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.494528 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.495752 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.496272 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.496365 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:54.499936 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:54.499946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:54.499959 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:54.499988 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:54.502757 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.503338 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.503344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.503346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.503450 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.503768 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.503771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.503773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.504054 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:54.504280 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:54.504600 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 15:16:54.504776 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":328,"threads_started":1}
I20260620 15:16:54.506198 6324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:54.588675 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 6201
I20260620 15:16:54.604152 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 6334
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[5961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:16:54.619118 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:54.605147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.0.97.67:43427)
0620 15:16:54.605743 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:54.605749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:54.605776 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:16:54.605834 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:54.605858 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:54.606061 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:54.606234 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:54.607281 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.608308 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.609537 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:54.610089 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:54.610227 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:54.614035 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:54.614045 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:54.614058 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:54.614087 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:54.616626 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.617263 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.617274 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.617284 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.617404 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:54.617750 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:54.617757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:54.617760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:54.618067 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:54.618369 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:54.618606 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 15:16:54.618840 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":312,"threads_started":1}
I20260620 15:16:54.620390 6457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:54.701789 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 6334
I20260620 15:16:54.718549 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 5974
I20260620 15:16:54.719784 6036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:54.818924 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 5974
2026-06-20T15:16:54Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3822 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[6489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[6489](info): set up 2 sockets
Jun 20 15:16:54 dist-test-slave-7s64 krb5kdc[6489](info): commencing operation
krb5kdc: starting...
W20260620 15:16:55.905154 389 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.009s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:16:56 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:16:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:16:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.126.keytab.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.126.keytab.
I20260620 15:16:56.098090 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46541
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:46541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:56.401346 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:56.401604 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:56.401687 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:56.408519 6502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:56.408601 6502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:56.408643 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:56.408699 6502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:56.408739 6502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:56.417851 6502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:46541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:56.418934 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:56.420056 6502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:56.431236 6508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:56.431422 6510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:56.431747 6507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:56.432053 6502 server_base.cc:1034] running on GCE node
I20260620 15:16:56.432890 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:56.434120 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:56.435352 6502 hybrid_clock.cc:648] HybridClock initialized: now 1781968616435312 us; error 45 us; skew 500 ppm
I20260620 15:16:56.435675 6502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:56 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:56.443859 6502 init.cc:377] Logged in from keytab as oryx/127.0.97.126@KRBTEST.COM (short username oryx)
I20260620 15:16:56.445288 6502 webserver.cc:460] Webserver started at http://127.0.97.126:36083/ using document root <none> and password file <none>
I20260620 15:16:56.445880 6502 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:56.445976 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:56.446197 6502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:56.448932 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "bca18fbf31a04a8ca99055a3367b1992"
format_stamp: "Formatted at 2026-06-20 15:16:56 on dist-test-slave-7s64"
server_key: "40bdb845fad57594ab499b32abb62710"
server_key_iv: "d253eaa3f7520d7c681e52cbf8c88e89"
server_key_version: "encryptionkey@0"
I20260620 15:16:56.449693 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "bca18fbf31a04a8ca99055a3367b1992"
format_stamp: "Formatted at 2026-06-20 15:16:56 on dist-test-slave-7s64"
server_key: "40bdb845fad57594ab499b32abb62710"
server_key_iv: "d253eaa3f7520d7c681e52cbf8c88e89"
server_key_version: "encryptionkey@0"
I20260620 15:16:56.454429 6502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:16:56.457520 6517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:56.458767 6502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:56.458920 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "bca18fbf31a04a8ca99055a3367b1992"
format_stamp: "Formatted at 2026-06-20 15:16:56 on dist-test-slave-7s64"
server_key: "40bdb845fad57594ab499b32abb62710"
server_key_iv: "d253eaa3f7520d7c681e52cbf8c88e89"
server_key_version: "encryptionkey@0"
I20260620 15:16:56.459090 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:56.472785 6502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:56.477101 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:56.477344 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:56.496611 6502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:46541
I20260620 15:16:56.496721 6568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:46541 every 8 connection(s)
I20260620 15:16:56.497989 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:16:56.501464 6569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:56.504156 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 6502
I20260620 15:16:56.504366 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:16:56.505198 389 external_mini_cluster.cc:1428] Setting key 6a97926fd0ff5fbe8163b118819c0d3a
I20260620 15:16:56.513787 6569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992: Bootstrap starting.
I20260620 15:16:56.517329 6569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:56.518594 6569 log.cc:824] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:16:56 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:16:56.521358 6569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992: No bootstrap required, opened a new log
I20260620 15:16:56.527870 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:56.507262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.0.97.126:46541)
0620 15:16:56.508165 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:56.508176 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:56.508218 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:16:56.509964 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:56.509985 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:56.510375 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:56.510698 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:56.512355 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:56.513715 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:56.515416 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:56.516087 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:56.516224 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:56.520958 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:56.520965 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:56.520978 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:56.521008 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:56.524129 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:56.524857 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:56.524863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:56.524868 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:56.525023 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:56.525381 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:56.525386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:56.525389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:56.525872 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:56.526194 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:56.526580 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 15:16:56.526960 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":427,"threads_started":1}
I20260620 15:16:56.529044 6569 raft_consensus.cc:348] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } }
I20260620 15:16:56.529381 6569 raft_consensus.cc:374] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:56.529496 6569 raft_consensus.cc:729] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca18fbf31a04a8ca99055a3367b1992, State: Initialized, Role: FOLLOWER
I20260620 15:16:56.530011 6569 consensus_queue.cc:260] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } }
I20260620 15:16:56.530221 6569 raft_consensus.cc:388] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:16:56.530376 6569 raft_consensus.cc:482] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:16:56.530546 6569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:56.532598 6569 raft_consensus.cc:504] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } }
I20260620 15:16:56.533051 6569 leader_election.cc:302] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bca18fbf31a04a8ca99055a3367b1992; no voters:
I20260620 15:16:56.533595 6569 leader_election.cc:288] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:16:56.533756 6574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:16:56.534966 6574 raft_consensus.cc:686] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 1 LEADER]: Becoming Leader. State: Replica: bca18fbf31a04a8ca99055a3367b1992, State: Running, Role: LEADER
I20260620 15:16:56.535596 6569 sys_catalog.cc:564] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:56.535557 6574 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } }
I20260620 15:16:56.538439 6576 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca18fbf31a04a8ca99055a3367b1992. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } }
I20260620 15:16:56.538472 6575 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } }
I20260620 15:16:56.538802 6576 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:56.538894 6575 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:56.545048 6581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:16:56.549587 6581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:16:56.558369 6581 catalog_manager.cc:1347] Generated new cluster ID: b14ee8ff09174c258de5f70796d4f926
I20260620 15:16:56.558480 6581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:16:56.568228 6581 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:16:56.569496 6581 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:16:56.576077 6581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992: Generated new TSK 0
I20260620 15:16:56.576841 6581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab.
I20260620 15:16:56.631350 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46541
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:56.960937 6595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:56.961176 6595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:56.961344 6595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:56.968284 6595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:56.968376 6595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:56.968592 6595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:56.977527 6595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:56.978633 6595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:56.979866 6595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:56.992349 6600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:56.992283 6601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:56.992199 6603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:56.993599 6595 server_base.cc:1034] running on GCE node
I20260620 15:16:56.994156 6595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:56.995023 6595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:56.996251 6595 hybrid_clock.cc:648] HybridClock initialized: now 1781968616996187 us; error 64 us; skew 500 ppm
I20260620 15:16:56.996628 6595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:56 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:56.999579 6595 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260620 15:16:57.000952 6595 webserver.cc:460] Webserver started at http://127.0.97.65:35343/ using document root <none> and password file <none>
I20260620 15:16:57.001528 6595 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:57.001621 6595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:57.001849 6595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:57.004454 6595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "22559783906942448d1eec02ed6331fe"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "09aa418c30fe6951ab0c4997239cc1a3"
server_key_iv: "cf3f61b274edeb21c9ec4eb9bba2a4b7"
server_key_version: "encryptionkey@0"
I20260620 15:16:57.005200 6595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "22559783906942448d1eec02ed6331fe"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "09aa418c30fe6951ab0c4997239cc1a3"
server_key_iv: "cf3f61b274edeb21c9ec4eb9bba2a4b7"
server_key_version: "encryptionkey@0"
I20260620 15:16:57.009836 6595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:16:57.013037 6610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:57.014351 6595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:16:57.014519 6595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "22559783906942448d1eec02ed6331fe"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "09aa418c30fe6951ab0c4997239cc1a3"
server_key_iv: "cf3f61b274edeb21c9ec4eb9bba2a4b7"
server_key_version: "encryptionkey@0"
I20260620 15:16:57.014711 6595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:57.040228 6595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:57.044128 6595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:57.044394 6595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:57.045301 6595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:57.046504 6595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:57.046573 6595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:57.046731 6595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:57.046789 6595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:57.078214 6595 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:33143
I20260620 15:16:57.078294 6722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:33143 every 8 connection(s)
I20260620 15:16:57.079396 6595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:57.082103 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 6595
I20260620 15:16:57.082314 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:16:57.083119 389 external_mini_cluster.cc:1428] Setting key 23806ba61ad4437b812663bd09b6eb89
Jun 20 15:16:57 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:16:57.099866 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:57.081946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:42949 (local address 127.0.97.126:46541)
0620 15:16:57.082404 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:57.082409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:57.083472 (+ 1063us) server_negotiation.cc:407] Connection header received
0620 15:16:57.085516 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:57.085521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:57.085601 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:57.085724 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:57.087103 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:57.087825 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:57.089416 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:57.089794 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:57.089932 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:57.094108 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:57.094112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:57.094116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:57.094145 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:57.096695 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:57.097588 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:57.097592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:57.097594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:57.097701 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:57.098202 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:57.098205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:57.098207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:57.098444 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:57.098556 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:57.099438 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 15:16:57.099609 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":236,"threads_started":1}
I20260620 15:16:57.101347 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:57.082602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.65:42949)
0620 15:16:57.083322 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:57.083363 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:57.085266 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:57.085953 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:57.085961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:57.086421 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:57.086937 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:57.086953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:57.087984 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:57.087989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:57.089255 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:57.089269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:57.090020 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:57.090027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:57.090246 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:57.090965 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:57.090997 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:57.093909 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:57.096821 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:57.096832 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:57.096851 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:57.097466 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:57.097815 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:57.097823 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:57.097829 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:57.098066 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:57.098568 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:57.098578 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:57.099031 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 15:16:57.099359 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 15:16:57.099944 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:16:57.102886 6723 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46541
I20260620 15:16:57.103204 6723 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:57.104007 6723 heartbeater.cc:511] Master 127.0.97.126:46541 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.66@KRBTEST.COM" created.
I20260620 15:16:57.106326 6534 ts_manager.cc:194] Registered new tserver with Master: 22559783906942448d1eec02ed6331fe (127.0.97.65:33143)
I20260620 15:16:57.107753 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:42949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab.
I20260620 15:16:57.128175 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46541
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:57.443986 6729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:57.444243 6729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:57.444342 6729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:57.451720 6729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:57.451833 6729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:57.451969 6729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:16:57.461535 6729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:57.462724 6729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:57.464141 6729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:57.477056 6735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:57.477234 6737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:57.477555 6734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:57.477880 6729 server_base.cc:1034] running on GCE node
I20260620 15:16:57.478418 6729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:57.479254 6729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:57.480466 6729 hybrid_clock.cc:648] HybridClock initialized: now 1781968617480435 us; error 38 us; skew 500 ppm
I20260620 15:16:57.480840 6729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:57 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:57.483886 6729 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260620 15:16:57.485323 6729 webserver.cc:460] Webserver started at http://127.0.97.66:43729/ using document root <none> and password file <none>
I20260620 15:16:57.485975 6729 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:57.486073 6729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:57.486367 6729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:57.489216 6729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "4bf8ff273da161a0306aeacc7ba3eee8"
server_key_iv: "9047c4c17b30091f2e826b1b144db632"
server_key_version: "encryptionkey@0"
I20260620 15:16:57.489967 6729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "4bf8ff273da161a0306aeacc7ba3eee8"
server_key_iv: "9047c4c17b30091f2e826b1b144db632"
server_key_version: "encryptionkey@0"
I20260620 15:16:57.494699 6729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:16:57.498152 6744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:57.499490 6729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 15:16:57.499694 6729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "4bf8ff273da161a0306aeacc7ba3eee8"
server_key_iv: "9047c4c17b30091f2e826b1b144db632"
server_key_version: "encryptionkey@0"
I20260620 15:16:57.499883 6729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:57.517989 6729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:57.521682 6729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:57.521958 6729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:57.522863 6729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:57.524065 6729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:57.524163 6729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:57.524304 6729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:57.524375 6729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:57.558172 6729 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37201
I20260620 15:16:57.558254 6856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37201 every 8 connection(s)
I20260620 15:16:57.559329 6729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:16:57.568074 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 6729
I20260620 15:16:57.568267 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:16:57.568826 389 external_mini_cluster.cc:1428] Setting key 61d2d50d178b4b8a1a40c0e65189c4c2
Jun 20 15:16:57 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:16:57.580159 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:57.561858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56577 (local address 127.0.97.126:46541)
0620 15:16:57.562084 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:57.562091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:57.563308 (+ 1217us) server_negotiation.cc:407] Connection header received
0620 15:16:57.565044 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:57.565049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:57.565117 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:57.565215 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:57.566576 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:57.567302 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:57.569535 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:57.570203 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:57.570340 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:57.574161 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:57.574166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:57.574169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:57.574199 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:57.577016 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:57.577708 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:57.577716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:57.577721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:57.577875 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:57.578286 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:57.578293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:57.578299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:57.578647 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:57.578789 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:57.579463 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 15:16:57.579717 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 15:16:57.580857 6859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:57.562398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.66:56577)
0620 15:16:57.563154 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:57.563194 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:57.564793 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:57.565408 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:57.565415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:57.565872 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:57.566385 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:57.566418 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:57.567472 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:57.567476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:57.569347 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:57.569363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:57.570416 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:57.570421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:57.570609 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:57.571339 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:57.571364 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:57.573956 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:57.577163 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:57.577169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:57.577180 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:57.577575 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:57.578004 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:57.578007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:57.578009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:57.578170 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:57.578793 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:57.578799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:57.579100 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 15:16:57.579336 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 15:16:57.579711 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":261,"threads_started":1}
I20260620 15:16:57.582365 6857 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46541
I20260620 15:16:57.582677 6857 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:57.583490 6857 heartbeater.cc:511] Master 127.0.97.126:46541 requested a full tablet report, sending...
I20260620 15:16:57.585018 6534 ts_manager.cc:194] Registered new tserver with Master: d3c3f0c9d7ff4262a9fab6dd8d53204b (127.0.97.66:37201)
I20260620 15:16:57.586186 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:56577
WARNING: no policy specified for oryx/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab.
I20260620 15:16:57.618211 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46541
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:57.930599 6862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:57.930857 6862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:57.930948 6862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:57.938038 6862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:57.938129 6862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:57.938233 6862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:16:57.947387 6862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:57.948427 6862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:57.949590 6862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:57.963893 6868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:57.964007 6867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:57.964394 6870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:57.966219 6862 server_base.cc:1034] running on GCE node
I20260620 15:16:57.966848 6862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:57.968115 6862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:57.969326 6862 hybrid_clock.cc:648] HybridClock initialized: now 1781968617969270 us; error 48 us; skew 500 ppm
I20260620 15:16:57.969709 6862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:57 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:57.974114 6862 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260620 15:16:57.975953 6862 webserver.cc:460] Webserver started at http://127.0.97.67:33081/ using document root <none> and password file <none>
I20260620 15:16:57.976867 6862 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:57.976994 6862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:57.977345 6862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:57.981601 6862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "fe55b5749b6843c899d0d2d8896c31cf"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "eb168d831ce778cd1bc44057811d63ce"
server_key_iv: "fd26d9497e380ff3405c22c63df0910f"
server_key_version: "encryptionkey@0"
I20260620 15:16:57.982628 6862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "fe55b5749b6843c899d0d2d8896c31cf"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "eb168d831ce778cd1bc44057811d63ce"
server_key_iv: "fd26d9497e380ff3405c22c63df0910f"
server_key_version: "encryptionkey@0"
I20260620 15:16:57.993499 6862 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.001s
I20260620 15:16:57.998433 6877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:57.999907 6862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:16:58.000092 6862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "fe55b5749b6843c899d0d2d8896c31cf"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "eb168d831ce778cd1bc44057811d63ce"
server_key_iv: "fd26d9497e380ff3405c22c63df0910f"
server_key_version: "encryptionkey@0"
I20260620 15:16:58.000273 6862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:58.018522 6862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:58.022619 6862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:58.022891 6862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:58.023818 6862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:58.025139 6862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:58.025239 6862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:58.025384 6862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:58.025463 6862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:58.058379 6862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39053
I20260620 15:16:58.058466 6989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39053 every 8 connection(s)
I20260620 15:16:58.059846 6862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:16:58.065099 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 6862
I20260620 15:16:58.065294 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:16:58.065703 389 external_mini_cluster.cc:1428] Setting key c13ca7a936cd52e731ee6a7dab3749e4
Jun 20 15:16:58 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968617, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:16:58.079531 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.062429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:40953 (local address 127.0.97.126:46541)
0620 15:16:58.062596 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:58.062600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:58.063805 (+ 1205us) server_negotiation.cc:407] Connection header received
0620 15:16:58.065691 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:58.065696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:58.065831 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:58.065961 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:58.067488 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.068707 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.070227 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.070621 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.070751 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.074116 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:58.074121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:58.074125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:58.074154 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:58.076466 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.077268 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.077274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.077280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.077409 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.077877 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.077882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.077886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.078202 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:58.078353 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:58.078962 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 15:16:58.079161 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:16:58.080579 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.062887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.67:40953)
0620 15:16:58.063657 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:58.063694 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:58.065433 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:58.066179 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:58.066187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:58.066742 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:58.067324 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.067341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.068861 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.068871 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.070084 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.070095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.070833 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.070841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.071072 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.071660 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:58.071684 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:58.073919 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:58.076641 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.076649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.076663 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.077052 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.077520 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.077525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.077529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.077748 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.078322 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:58.078327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:58.078599 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 15:16:58.078808 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 15:16:58.079309 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":297,"threads_started":1}
I20260620 15:16:58.082031 6990 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46541
I20260620 15:16:58.082322 6990 heartbeater.cc:464] Registering TS with master...
I20260620 15:16:58.083128 6990 heartbeater.cc:511] Master 127.0.97.126:46541 requested a full tablet report, sending...
I20260620 15:16:58.084529 6534 ts_manager.cc:194] Registered new tserver with Master: fe55b5749b6843c899d0d2d8896c31cf (127.0.97.67:39053)
I20260620 15:16:58.085282 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:40953
I20260620 15:16:58.093071 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:16:58.110916 6723 heartbeater.cc:503] Master 127.0.97.126:46541 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.125.keytab.
Entry for principal oryx/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.125.keytab.
I20260620 15:16:58.135641 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:37799
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.125.keytab
--principal=oryx/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:46541,127.0.97.125:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:16:58.442683 6995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:58.442986 6995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:58.443149 6995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:58.450325 6995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:58.450470 6995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:58.450567 6995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:58.450663 6995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:58.450745 6995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:58.460222 6995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:46541,127.0.97.125:37799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.125.keytab
--principal=oryx/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
W20260620 15:16:58.460582 6995 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.126:46541,127.0.97.125:37799'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 15:16:58.461534 6995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:58.462843 6995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:58.473984 7000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:58.474012 7001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:58.473938 7003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:58.474869 6995 server_base.cc:1034] running on GCE node
I20260620 15:16:58.475585 6995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:58.476922 6995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:58.478171 6995 hybrid_clock.cc:648] HybridClock initialized: now 1781968618478109 us; error 43 us; skew 500 ppm
I20260620 15:16:58.478485 6995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:58 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:58.481603 6995 init.cc:377] Logged in from keytab as oryx/127.0.97.125@KRBTEST.COM (short username oryx)
I20260620 15:16:58.483034 6995 webserver.cc:460] Webserver started at http://127.0.97.125:43149/ using document root <none> and password file <none>
I20260620 15:16:58.483675 6995 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:58.483780 6995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:58.484082 6995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:58.487064 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data/instance:
uuid: "0c419a33235d428ba328b43e6fe45a1e"
format_stamp: "Formatted at 2026-06-20 15:16:58 on dist-test-slave-7s64"
server_key: "305d9306151d8f47f9381c647d7e660e"
server_key_iv: "9c291cf65a1c4050ae3edac9fd0a3764"
server_key_version: "encryptionkey@0"
I20260620 15:16:58.487809 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal/instance:
uuid: "0c419a33235d428ba328b43e6fe45a1e"
format_stamp: "Formatted at 2026-06-20 15:16:58 on dist-test-slave-7s64"
server_key: "305d9306151d8f47f9381c647d7e660e"
server_key_iv: "9c291cf65a1c4050ae3edac9fd0a3764"
server_key_version: "encryptionkey@0"
I20260620 15:16:58.492480 6995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 15:16:58.495666 7010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:58.497128 6995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:58.497269 6995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal
uuid: "0c419a33235d428ba328b43e6fe45a1e"
format_stamp: "Formatted at 2026-06-20 15:16:58 on dist-test-slave-7s64"
server_key: "305d9306151d8f47f9381c647d7e660e"
server_key_iv: "9c291cf65a1c4050ae3edac9fd0a3764"
server_key_version: "encryptionkey@0"
I20260620 15:16:58.497481 6995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:58.521808 6995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:58.525786 6995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:58.525991 6995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:58.543154 6995 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:37799
I20260620 15:16:58.543181 7061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:37799 every 8 connection(s)
I20260620 15:16:58.544450 6995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
I20260620 15:16:58.547905 7062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:58.552039 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 6995
I20260620 15:16:58.555087 7062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:16:58 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
Jun 20 15:16:58 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260620 15:16:58.575709 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.560316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.0.97.126:46541)
0620 15:16:58.560538 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:58.560547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:58.561058 (+ 511us) server_negotiation.cc:407] Connection header received
0620 15:16:58.561322 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:58.561347 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:58.561433 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:58.561532 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:58.562997 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.563866 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.565480 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.565904 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.566045 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.569693 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:58.569698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:58.569702 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:58.569730 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:58.572479 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.573652 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.573655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.573658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.573763 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.574200 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.574203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.574205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.574462 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:58.574614 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:58.575197 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 15:16:58.575383 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:16:58.576856 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.560261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:33300)
0620 15:16:58.560922 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:58.560960 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:58.561155 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:58.561616 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:58.561622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:58.562233 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:58.562802 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.562828 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.564014 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.564021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.565313 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.565325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.566144 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.566152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.566321 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.567120 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:58.567148 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:58.569523 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:58.572681 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.572692 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.572703 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.573527 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.573880 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.573885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.573889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.574096 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.574606 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:58.574613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:58.574870 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 15:16:58.575116 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 15:16:58.575530 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":323,"threads_started":1}
I20260620 15:16:58.576889 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.554185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.0.97.125:37799)
0620 15:16:58.554957 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:58.554973 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:58.555019 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:16:58.558448 (+ 3429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:58.558476 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:58.558984 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:58.559370 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:58.562556 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.564233 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.565476 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.565957 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.566167 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.570301 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:58.570309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:58.570319 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:58.570361 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:58.573557 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.574354 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.574362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.574365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.574499 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.574898 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.574903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.574907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.575216 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:58.575539 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:58.575881 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 15:16:58.576118 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":340,"threads_started":1}
I20260620 15:16:58.578543 7062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:16:58 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260620 15:16:58.589329 6857 heartbeater.cc:503] Master 127.0.97.126:46541 was elected leader, sending a full tablet report...
I20260620 15:16:58.589723 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.579271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:37799 (local address 127.0.0.1:59658)
0620 15:16:58.579430 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:58.579598 (+ 168us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:58.579668 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:58.579934 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:58.579936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:58.580195 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:58.580302 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.580308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.581290 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.581297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.582190 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.582201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.582823 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.582827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.582962 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.583410 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:58.583425 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:58.585160 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:58.587593 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.587596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.587599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.587907 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.588253 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.588256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.588258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.588362 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.588916 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:58.588919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:58.589024 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 15:16:58.589177 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 15:16:58.589432 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 15:16:58.590319 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.579496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.0.97.125:37799)
0620 15:16:58.579654 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:58.579658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:58.579675 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:16:58.579737 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:58.579740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:58.579788 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:58.579896 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:58.580410 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.581150 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.582316 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.582707 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.582809 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.585338 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:58.585342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:58.585345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:58.585374 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:58.587483 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.588027 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.588031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.588033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.588154 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.588477 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.588481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.588483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.588809 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:58.588936 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:58.589306 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 15:16:58.590047 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":47}
I20260620 15:16:58.595153 7062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: Bootstrap starting.
I20260620 15:16:58.598274 7062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:58.599303 7062 log.cc:824] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:58.601918 7062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: No bootstrap required, opened a new log
I20260620 15:16:58.609434 7062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } }
I20260620 15:16:58.609761 7062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:58.609884 7062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c419a33235d428ba328b43e6fe45a1e, State: Initialized, Role: FOLLOWER
I20260620 15:16:58.610430 7062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } }
I20260620 15:16:58.611167 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } } }
I20260620 15:16:58.611426 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:16:58.612206 7062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:16:58.620136 7079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:16:58.620267 7079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:16:58.622568 6995 master_runner.cc:428] Detected that this master 0c419a33235d428ba328b43e6fe45a1e is joining an existing cluster
I20260620 15:16:58.622653 6995 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.125:37799
I20260620 15:16:58.642513 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.626059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33314 (local address 127.0.97.126:46541)
0620 15:16:58.626237 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:58.626245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:58.627149 (+ 904us) server_negotiation.cc:407] Connection header received
0620 15:16:58.627355 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:58.627360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:58.627448 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:58.627547 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:58.629102 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.630201 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.632191 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.632888 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.632981 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.635931 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:58.635938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:58.635943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:58.635979 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:58.639399 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.640194 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.640199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.640205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.640345 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.640846 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.640851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.640854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.641178 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:58.641291 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:58.641933 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 15:16:58.642148 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:16:58.643563 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.626072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:33314)
0620 15:16:58.627022 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:58.627047 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:58.627209 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:58.627603 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:58.627611 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:58.628400 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:58.628930 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.628945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.630360 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.630365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.632034 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.632049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.633249 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.633255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.633809 (+ 554us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.634548 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:58.634581 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:58.635709 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:58.639554 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.639560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.639564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.640047 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.640524 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.640533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.640537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.640712 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.641398 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:58.641406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:58.641628 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 15:16:58.642781 (+ 1153us) client_negotiation.cc:240] Negotiation successful
0620 15:16:58.643098 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":597,"threads_started":1}
I20260620 15:16:58.649295 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.626419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:37799 (local address 127.0.0.1:59674)
0620 15:16:58.633186 (+ 6767us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:58.633212 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:58.633358 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:58.633682 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:58.633689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:58.634394 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:58.634777 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.634794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.636305 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.636313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.638084 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.638100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.639404 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.639411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.639629 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.640234 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:58.640262 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:58.641429 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:58.646149 (+ 4720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.646157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.646163 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.646701 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.647265 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.647272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.647278 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.647454 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.648109 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:58.648116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:58.648261 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 15:16:58.648399 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 15:16:58.648739 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6611,"thread_start_us":7001,"threads_started":1}
I20260620 15:16:58.649319 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.626025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.0.97.125:37799)
0620 15:16:58.626517 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:58.626523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:58.633279 (+ 6756us) server_negotiation.cc:407] Connection header received
0620 15:16:58.633451 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:58.633456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:58.633521 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:58.633612 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:58.634929 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.636038 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.638227 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.639014 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.639127 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.641628 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:58.641634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:58.641642 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:58.641678 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:58.644844 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.647000 (+ 2156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.647005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.647008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.647142 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.647563 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.647570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.647576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.647910 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:58.648046 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:58.648558 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 15:16:58.648906 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":309}
WARNING: no policy specified for oryx/127.0.97.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.124@KRBTEST.COM" created.
Jun 20 15:16:58 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260620 15:16:58.667614 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.649772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59678 (local address 127.0.97.125:37799)
0620 15:16:58.649944 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:58.649951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:58.651575 (+ 1624us) server_negotiation.cc:407] Connection header received
0620 15:16:58.651808 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:58.651815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:58.651920 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:58.652020 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:58.653055 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.653733 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.654943 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.655405 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.655535 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.660831 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:58.660839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:58.660848 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:58.660892 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:58.664836 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.665625 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.665632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.665637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.665793 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.666207 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.666213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.666218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.666474 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:58.666622 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:58.667019 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 15:16:58.667235 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:16:58.667953 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.649285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:37799 (local address 127.0.0.1:59678)
0620 15:16:58.651431 (+ 2146us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:58.651470 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:58.651626 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:58.652074 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:58.652079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:58.652727 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:58.652912 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.652920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.653853 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.653859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.654810 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.654821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.655557 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.655563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.655756 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.656403 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:58.656423 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:58.660628 (+ 4205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:58.664965 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.664976 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.664983 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.665493 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.665922 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.665927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.665931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.666093 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.666606 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:58.666613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:58.666763 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 15:16:58.666932 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 15:16:58.667242 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1969,"thread_start_us":1836,"threads_started":1}
I20260620 15:16:58.669780 6534 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.125:37799: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.125"
port: 37799
}
attrs {
promote: true
}
}
dest_uuid: "bca18fbf31a04a8ca99055a3367b1992"
cas_config_opid_index: -1
I20260620 15:16:58.670658 6534 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } }
I20260620 15:16:58.680435 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0c419a33235d428ba328b43e6fe45a1e (127.0.97.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } }
I20260620 15:16:58.681514 7037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:58.684052 7037 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Refusing update from remote peer bca18fbf31a04a8ca99055a3367b1992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 15:16:58.685056 7092 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.125:37799
I20260620 15:16:58.685581 7091 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } }
I20260620 15:16:58.685856 7091 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:58.685940 6995 master.cc:537] Master@127.0.97.125:37799 shutting down...
I20260620 15:16:58.687071 7092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.002s
W20260620 15:16:58.690044 6520 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.124.keytab.
Entry for principal oryx/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.124.keytab.
W20260620 15:16:58.691725 6520 consensus_peers.cc:487] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 -> Peer 0c419a33235d428ba328b43e6fe45a1e (127.0.97.125:37799): Couldn't send request to peer 0c419a33235d428ba328b43e6fe45a1e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 15:16:58.692322 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:42865
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.124.keytab
--principal=oryx/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:46541,127.0.97.125:37799,127.0.97.124:42865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:16:58.697121 7068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:58.697355 7068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } }
I20260620 15:16:58.698452 7068 leader_election.cc:288] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bca18fbf31a04a8ca99055a3367b1992 (127.0.97.126:46541)
I20260620 15:16:58.699028 6544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c419a33235d428ba328b43e6fe45a1e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca18fbf31a04a8ca99055a3367b1992" is_pre_election: true
I20260620 15:16:58.700021 7011 leader_election.cc:302] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0c419a33235d428ba328b43e6fe45a1e; no voters: bca18fbf31a04a8ca99055a3367b1992
I20260620 15:16:58.700685 6995 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:16:58.700910 7068 raft_consensus.cc:2684] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Received pre-election callback for term 2 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260620 15:16:58.701191 6995 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:16:58.701413 6995 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: stopping tablet replica
I20260620 15:16:58.708031 6995 master.cc:552] Master@127.0.97.125:37799 shutdown complete.
I20260620 15:16:58.708108 6995 master_runner.cc:305] Clearing existing system tablet
I20260620 15:16:58.708557 6995 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:58.708647 6995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:58.711504 6995 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 15:16:58.719143 7098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:58.719511 6995 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:58.719798 6995 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.006s
I20260620 15:16:58.719878 6995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal
uuid: "0c419a33235d428ba328b43e6fe45a1e"
format_stamp: "Formatted at 2026-06-20 15:16:58 on dist-test-slave-7s64"
server_key: "305d9306151d8f47f9381c647d7e660e"
server_key_iv: "9c291cf65a1c4050ae3edac9fd0a3764"
server_key_version: "encryptionkey@0"
I20260620 15:16:58.724859 6995 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:58.731412 6995 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 15:16:58.731562 6995 log.cc:1196] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 15:16:58.731952 6995 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: Deleting consensus metadata
I20260620 15:16:58.732194 6995 master_runner.cc:315] Copying system tablet from 127.0.97.126:46541
I20260620 15:16:58.734575 6995 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.126:46541
I20260620 15:16:58.754980 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.735320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.0.97.126:46541)
0620 15:16:58.735503 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:58.735509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:58.741073 (+ 5564us) server_negotiation.cc:407] Connection header received
0620 15:16:58.741239 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:58.741245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:58.741340 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:58.741455 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:58.742437 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.743524 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.745413 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.746276 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.746427 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.748648 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:58.748656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:58.748664 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:58.748708 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:58.752205 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.752973 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.752978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.752982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.753124 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:58.753552 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:58.753557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:58.753560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:58.753913 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:58.754071 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:58.754465 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 15:16:58.754676 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:16:58.755086 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:58.735252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:33326)
0620 15:16:58.740926 (+ 5674us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:58.740952 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:58.741104 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:58.741492 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:58.741498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:58.741998 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:58.742271 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.742283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.743671 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.743677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.745226 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:58.745241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:58.746443 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:58.746449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:58.746629 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:58.747377 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:58.747401 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:58.748426 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:58.752340 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.752354 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.752358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.752841 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.753247 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:58.753252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:58.753255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:58.753433 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:58.754043 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:58.754048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:58.754216 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 15:16:58.754366 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 15:16:58.754637 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5504,"thread_start_us":5373,"threads_started":1}
I20260620 15:16:58.756109 6554 tablet_copy_service.cc:140] P bca18fbf31a04a8ca99055a3367b1992: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0c419a33235d428ba328b43e6fe45a1e ({username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:33326)
I20260620 15:16:58.756346 6554 tablet_copy_service.cc:161] P bca18fbf31a04a8ca99055a3367b1992: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0c419a33235d428ba328b43e6fe45a1e at {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:33326: session id = 0c419a33235d428ba328b43e6fe45a1e-00000000000000000000000000000000
I20260620 15:16:58.760608 6554 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 15:16:58.762710 6995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 15:16:58.769317 6995 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: tablet copy: Starting download of 0 data blocks...
I20260620 15:16:58.769611 6995 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: tablet copy: Starting download of 1 WAL segments...
I20260620 15:16:58.772148 6995 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 15:16:58.776269 6554 tablet_copy_service.cc:342] P bca18fbf31a04a8ca99055a3367b1992: Request end of tablet copy session 0c419a33235d428ba328b43e6fe45a1e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:33326
I20260620 15:16:58.776460 6554 tablet_copy_service.cc:433] P bca18fbf31a04a8ca99055a3367b1992: ending tablet copy session 0c419a33235d428ba328b43e6fe45a1e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0c419a33235d428ba328b43e6fe45a1e
I20260620 15:16:58.787139 6995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:58.789738 7105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:58.790171 7106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:58.800194 7108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:58.801635 6995 server_base.cc:1034] running on GCE node
I20260620 15:16:58.801964 6995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:58.802640 6995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:58.803848 6995 hybrid_clock.cc:648] HybridClock initialized: now 1781968618803750 us; error 70 us; skew 500 ppm
I20260620 15:16:58.804220 6995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:58 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968618, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:58.806313 6995 init.cc:377] Logged in from keytab as oryx/127.0.97.125@KRBTEST.COM (short username oryx)
I20260620 15:16:58.807425 6995 webserver.cc:460] Webserver started at http://127.0.97.125:44483/ using document root <none> and password file <none>
I20260620 15:16:58.807745 6995 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:58.807830 6995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:58.810792 6995 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:16:58.812582 7115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:58.812966 6995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 15:16:58.813144 6995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal
uuid: "0c419a33235d428ba328b43e6fe45a1e"
format_stamp: "Formatted at 2026-06-20 15:16:58 on dist-test-slave-7s64"
server_key: "305d9306151d8f47f9381c647d7e660e"
server_key_iv: "9c291cf65a1c4050ae3edac9fd0a3764"
server_key_version: "encryptionkey@0"
I20260620 15:16:58.813357 6995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:58.843756 6995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:58.848117 6995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:58.867451 6995 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:37799
I20260620 15:16:58.867508 7166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:37799 every 8 connection(s)
I20260620 15:16:58.868599 6995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
I20260620 15:16:58.874248 7167 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:16:58.877982 7167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: Bootstrap starting.
I20260620 15:16:58.893379 7167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:16:58.893889 7167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: Bootstrap complete.
I20260620 15:16:58.894871 7167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } }
I20260620 15:16:58.895124 7167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0c419a33235d428ba328b43e6fe45a1e, State: Initialized, Role: LEARNER
I20260620 15:16:58.895360 7167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } }
I20260620 15:16:58.896410 7167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:58.896175 7170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } }
I20260620 15:16:58.896720 7170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: This master's current role is: LEARNER
I20260620 15:16:58.908409 7181 catalog_manager.cc:1259] Loaded cluster ID: b14ee8ff09174c258de5f70796d4f926
I20260620 15:16:58.908532 7181 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: loading cluster ID for follower catalog manager: success
I20260620 15:16:58.912205 7181 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: acquiring CA information for follower catalog manager: success
I20260620 15:16:58.914593 7181 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 15:16:59.065282 7096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:59.065519 7096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:59.065673 7096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:59.072705 7096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:16:59.072788 7096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:59.072885 7096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:59.072978 7096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:16:59.073062 7096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:16:59.082839 7096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:46541,127.0.97.125:37799,127.0.97.124:42865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.124.keytab
--principal=oryx/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:42865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:59.083945 7096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:59.085306 7096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:16:59.087471 6990 heartbeater.cc:503] Master 127.0.97.126:46541 was elected leader, sending a full tablet report...
W20260620 15:16:59.097139 7189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:59.097206 7186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:59.097483 7187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:59.098310 7096 server_base.cc:1034] running on GCE node
I20260620 15:16:59.098986 7096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:59.100327 7096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:59.101564 7096 hybrid_clock.cc:648] HybridClock initialized: now 1781968619101511 us; error 46 us; skew 500 ppm
I20260620 15:16:59.101873 7096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:59.105625 7096 init.cc:377] Logged in from keytab as oryx/127.0.97.124@KRBTEST.COM (short username oryx)
I20260620 15:16:59.107018 7096 webserver.cc:460] Webserver started at http://127.0.97.124:41431/ using document root <none> and password file <none>
I20260620 15:16:59.107638 7096 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:59.107767 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:59.108059 7096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:16:59.109386 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.102747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:37799 (local address 127.0.0.1:59680)
0620 15:16:59.102893 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.102916 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.103002 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.104154 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.104159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.104191 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:59.104379 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.104394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.105897 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.105901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.107395 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.107407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.108645 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.108651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.108838 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.108895 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.108997 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.109122 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 15:16:59.109426 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.103321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.0.97.125:37799)
0620 15:16:59.103787 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.103792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.103811 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:16:59.103895 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.103902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.103982 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.104072 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:59.104598 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.105682 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.107541 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.108458 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.108626 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.108834 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.109020 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.109138 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260620 15:16:59.110773 7142 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 15:16:59.111275 7170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca18fbf31a04a8ca99055a3367b1992. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } }
I20260620 15:16:59.111511 7170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: This master's current role is: LEARNER
I20260620 15:16:59.111634 7096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data/instance:
uuid: "eb7bb01e759840caa7c9db33aaefe281"
format_stamp: "Formatted at 2026-06-20 15:16:59 on dist-test-slave-7s64"
server_key: "5446ef653f41e71050d2780ac55dfd9d"
server_key_iv: "dfa548c93ce41f663b17374cc61bc8b1"
server_key_version: "encryptionkey@0"
I20260620 15:16:59.112339 7096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal/instance:
uuid: "eb7bb01e759840caa7c9db33aaefe281"
format_stamp: "Formatted at 2026-06-20 15:16:59 on dist-test-slave-7s64"
server_key: "5446ef653f41e71050d2780ac55dfd9d"
server_key_iv: "dfa548c93ce41f663b17374cc61bc8b1"
server_key_version: "encryptionkey@0"
I20260620 15:16:59.117267 7096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 15:16:59.120604 7197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:59.121830 7096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:16:59.121992 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal
uuid: "eb7bb01e759840caa7c9db33aaefe281"
format_stamp: "Formatted at 2026-06-20 15:16:59 on dist-test-slave-7s64"
server_key: "5446ef653f41e71050d2780ac55dfd9d"
server_key_iv: "dfa548c93ce41f663b17374cc61bc8b1"
server_key_version: "encryptionkey@0"
I20260620 15:16:59.122164 7096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:59.137686 7096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:59.142781 7096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:59.143072 7096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:59.161726 7096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:42865
I20260620 15:16:59.161798 7248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:42865 every 8 connection(s)
I20260620 15:16:59.162997 7096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
I20260620 15:16:59.165825 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 7096
I20260620 15:16:59.167644 7249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:16:59.176411 7249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:16:59.188820 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.168002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.0.97.124:42865)
0620 15:16:59.168782 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.168794 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.168832 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:16:59.170306 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.170325 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.170689 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.171100 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.172817 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.174091 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.176030 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.176786 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.176922 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.181999 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.182006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.182014 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.182045 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.185007 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.185648 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.185656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.185665 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.185823 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.186187 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.186194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.186197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.186710 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.187012 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.187442 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.187821 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":309,"threads_started":1}
I20260620 15:16:59.192690 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.179263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.0.97.126:46541)
0620 15:16:59.179458 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.179464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.179894 (+ 430us) server_negotiation.cc:407] Connection header received
0620 15:16:59.180061 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.180068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.180159 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.180249 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.181632 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.182320 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.183676 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.184108 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.184244 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.187278 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.187295 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.187298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.187329 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.190260 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.190899 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.190906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.190910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.191032 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.191405 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.191410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.191413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.191696 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.191802 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.192236 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.192395 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:16:59.192723 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.179099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:33330)
0620 15:16:59.179761 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.179799 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.179936 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.180343 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.180350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.181157 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.181482 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.181495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.182458 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.182464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.183527 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.183539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.184281 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.184287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.184441 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.185057 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.185076 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.187096 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.190383 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.190394 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.190401 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.190785 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.191137 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.191142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.191146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.191274 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.191812 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.191818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.191937 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.192148 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.192394 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":288,"threads_started":1}
I20260620 15:16:59.194006 7249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260620 15:16:59.205615 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.194648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:37799 (local address 127.0.0.1:59692)
0620 15:16:59.194845 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.194864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.194953 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.195278 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.195283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.195592 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.195705 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.195712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.196696 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.196701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.197576 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.197586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.198321 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.198327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.198500 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.198966 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.198992 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.200873 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.203584 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.203591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.203595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.203934 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.204312 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.204318 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.204322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.204449 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.205032 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.205035 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.205137 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.205243 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.205416 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 15:16:59.205821 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.194721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.0.97.125:37799)
0620 15:16:59.194924 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.194930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.194947 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:16:59.195100 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.195104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.195165 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.195263 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.195837 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.196565 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.197760 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.198185 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.198312 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.201070 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.201074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.201077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.201107 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.203460 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.204069 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.204077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.204082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.204208 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.204610 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.204614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.204616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.204861 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.205055 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.205360 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.205548 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:16:59.206768 7249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
I20260620 15:16:59.218386 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.207296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:42865 (local address 127.0.0.1:56492)
0620 15:16:59.207463 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.207477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.207546 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.207778 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.207781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.208073 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.208189 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.208196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.209208 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.209212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.209980 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.210000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.210606 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.210609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.210726 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.211158 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.211173 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.213709 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.216231 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.216238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.216243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.216687 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.217031 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.217037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.217042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.217179 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.217705 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.217709 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.217846 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.217978 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.218135 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 15:16:59.218642 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.207450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.0.97.124:42865)
0620 15:16:59.207582 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.207586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.207599 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:16:59.207633 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.207636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.207680 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.207750 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.208319 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.209098 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.210100 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.210483 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.210572 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.213872 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.213877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.213882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.213917 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.216117 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.216791 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.216799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.216802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.216914 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.217283 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.217288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.217291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.217525 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.217683 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.218082 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.218244 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 15:16:59.223054 7249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Bootstrap starting.
I20260620 15:16:59.226300 7249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Neither blocks nor log segments found. Creating new log.
I20260620 15:16:59.227267 7249 log.cc:824] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Log is configured to *not* fsync() on all Append() calls
I20260620 15:16:59.229804 7249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: No bootstrap required, opened a new log
I20260620 15:16:59.236711 7249 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } }
I20260620 15:16:59.236989 7249 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:16:59.237080 7249 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7bb01e759840caa7c9db33aaefe281, State: Initialized, Role: FOLLOWER
I20260620 15:16:59.237562 7249 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } }
I20260620 15:16:59.238322 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } } }
I20260620 15:16:59.238597 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:16:59.239239 7249 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:16:59.248932 7266 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:16:59.249065 7266 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:16:59.252887 7096 master_runner.cc:428] Detected that this master eb7bb01e759840caa7c9db33aaefe281 is joining an existing cluster
I20260620 15:16:59.253029 7096 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.124:42865
I20260620 15:16:59.254244 7255 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:16:59.254550 7255 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } }
I20260620 15:16:59.256325 7255 leader_election.cc:288] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bca18fbf31a04a8ca99055a3367b1992 (127.0.97.126:46541), 0c419a33235d428ba328b43e6fe45a1e (127.0.97.125:37799)
I20260620 15:16:59.256773 6544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb7bb01e759840caa7c9db33aaefe281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca18fbf31a04a8ca99055a3367b1992" is_pre_election: true
I20260620 15:16:59.257632 7142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb7bb01e759840caa7c9db33aaefe281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c419a33235d428ba328b43e6fe45a1e" is_pre_election: true
I20260620 15:16:59.258419 7200 leader_election.cc:302] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb7bb01e759840caa7c9db33aaefe281; no voters: 0c419a33235d428ba328b43e6fe45a1e, bca18fbf31a04a8ca99055a3367b1992
I20260620 15:16:59.258782 7255 raft_consensus.cc:2726] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 15:16:59.276517 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.259315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.0.97.126:46541)
0620 15:16:59.259502 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.259521 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.260507 (+ 986us) server_negotiation.cc:407] Connection header received
0620 15:16:59.260808 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.260816 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.260915 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.261009 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.263294 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.264446 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.266582 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.267212 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.267322 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.270130 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.270136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.270141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.270176 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.273575 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.274464 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.274470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.274474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.274599 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.275027 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.275050 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.275054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.275354 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.275446 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.275901 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.276120 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:16:59.276641 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.259454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:33332)
0620 15:16:59.260374 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.260399 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.260633 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.261243 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.261250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.262605 (+ 1355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.263082 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.263099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.264695 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.264705 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.266417 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.266432 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.267579 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.267597 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.267766 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.268463 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.268505 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.269557 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.273717 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.273727 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.273731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.274330 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.274732 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.274736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.274740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.274900 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.275516 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.275519 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.275670 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.275822 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.276158 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":443,"threads_started":1}
I20260620 15:16:59.280481 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.260693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:42865 (local address 127.0.0.1:56498)
0620 15:16:59.264962 (+ 4269us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.264980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.265239 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.265648 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.265652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.266319 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.266652 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.266662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.268114 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.268118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.270034 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.270044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.271223 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.271228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.271399 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.272075 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.272102 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.273153 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.277144 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.277149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.277152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.277614 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.277995 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.277998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.278002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.278137 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.278783 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.278787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.279500 (+ 713us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.279674 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.279909 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4093,"spinlock_wait_cycles":17280,"thread_start_us":4371,"threads_started":1}
I20260620 15:16:59.281378 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.259394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59706 (local address 127.0.97.125:37799)
0620 15:16:59.259603 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.259608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.263708 (+ 4100us) server_negotiation.cc:407] Connection header received
0620 15:16:59.263833 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.263839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.263909 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.263995 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.265442 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.266432 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.268507 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.269121 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.269211 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.272237 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.272243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.272247 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.272284 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.277705 (+ 5421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.278443 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.278448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.278451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.278582 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.279468 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.279474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.279477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.280073 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.280201 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.280814 (+ 613us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.281036 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:16:59.281610 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.260412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.0.97.124:42865)
0620 15:16:59.260593 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.260598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.265053 (+ 4455us) server_negotiation.cc:407] Connection header received
0620 15:16:59.265361 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.265365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.265423 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.265487 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.266802 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.267928 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.270139 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.270868 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.270987 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.273320 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.273326 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.273331 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.273364 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.276796 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.277717 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.277723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.277725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.277877 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.278243 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.278248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.278250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.278601 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.278722 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.281102 (+ 2380us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.281295 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 15:16:59.282291 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.259535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:37799 (local address 127.0.0.1:59706)
0620 15:16:59.263609 (+ 4074us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.263627 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.263736 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.264150 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.264155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.264977 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.265272 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.265282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.266620 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.266624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.268299 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.268313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.270029 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.270034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.270211 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.270967 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.270993 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.272044 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.277877 (+ 5833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.277882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.277886 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.278310 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.278720 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.278724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.278728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.279334 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.280309 (+ 975us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.280314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.280530 (+ 216us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.281539 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.281820 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3953,"spinlock_wait_cycles":21888,"thread_start_us":3778,"threads_started":1}
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
I20260620 15:16:59.294672 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 6595
I20260620 15:16:59.295500 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.281723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:42865 (local address 127.0.0.1:56508)
0620 15:16:59.282049 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.282068 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.282158 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.283571 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.283575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.283902 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.284054 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.284062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.285368 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.285376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.286211 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.286221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.286947 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.286953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.287115 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.287664 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.287679 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.289897 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.293026 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.293033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.293037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.293451 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.293849 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.293853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.293856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.293999 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.294584 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.294589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.294740 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.294898 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.295159 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260620 15:16:59.295617 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.282112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.0.97.124:42865)
0620 15:16:59.283244 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.283249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.283267 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:16:59.283336 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.283340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.283418 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.284159 (+ 741us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.284379 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.285247 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.286340 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.286786 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.286924 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.290115 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.290119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.290122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.290147 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.292884 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.293570 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.293580 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.293584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.293731 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.294116 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.294121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.294125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.294452 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.294598 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.295001 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.295242 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":880,"threads_started":1}
I20260620 15:16:59.297145 6534 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.124:42865: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.124"
port: 42865
}
attrs {
promote: true
}
}
dest_uuid: "bca18fbf31a04a8ca99055a3367b1992"
cas_config_opid_index: 5
I20260620 15:16:59.298177 6534 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } }
I20260620 15:16:59.299983 7092 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER eb7bb01e759840caa7c9db33aaefe281 (127.0.97.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.303891 7142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 LEARNER]: Refusing update from remote peer bca18fbf31a04a8ca99055a3367b1992: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 15:16:59.305277 7224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:16:59.308648 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.65@KRBTEST.COM
I20260620 15:16:59.309520 7224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 FOLLOWER]: Refusing update from remote peer bca18fbf31a04a8ca99055a3367b1992: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 15:16:59.319514 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.296385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.0.97.65:33143)
0620 15:16:59.297299 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.297308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.297339 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:16:59.297409 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.297417 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.297676 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.297847 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.299387 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.301006 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.303362 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.304242 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.310115 (+ 5873us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.310359 (+ 244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.310369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.310384 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.310412 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.314888 (+ 4476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.315947 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.315960 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.315966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.316148 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.316588 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.316593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.316596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.316966 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.317272 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.318584 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.318907 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":14678,"threads_started":1}
I20260620 15:16:59.320791 6718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:16:59.323418 7142 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER eb7bb01e759840caa7c9db33aaefe281 (127.0.97.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.326848 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:16:59.337314 7092 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.124:42865
I20260620 15:16:59.338300 7096 master.cc:537] Master@127.0.97.124:42865 shutting down...
I20260620 15:16:59.338565 7224 raft_consensus.cc:858] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } }
I20260620 15:16:59.339448 7170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.339745 7170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: This master's current role is: LEARNER
W20260620 15:16:59.340179 7224 raft_consensus.cc:1565] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } }
I20260620 15:16:59.340054 7279 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.342643 7279 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:59.342954 7092 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992: attempting to promote NON_VOTER 0c419a33235d428ba328b43e6fe45a1e to VOTER
I20260620 15:16:59.343695 7092 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } }
I20260620 15:16:59.348644 7142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 LEARNER]: Refusing update from remote peer bca18fbf31a04a8ca99055a3367b1992: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260620 15:16:59.348838 6518 consensus_peers.cc:487] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 -> Peer eb7bb01e759840caa7c9db33aaefe281 (127.0.97.124:42865): Couldn't send request to peer eb7bb01e759840caa7c9db33aaefe281. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 15:16:59.349243 7278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 15:16:59.352039 7278 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0c419a33235d428ba328b43e6fe45a1e (127.0.97.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.357374 7142 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0c419a33235d428ba328b43e6fe45a1e (127.0.97.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.361801 7170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.362028 7170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:16:59.363307 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.363571 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:59.363857 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.364069 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: This master's current role is: LEADER
I20260620 15:16:59.368762 7224 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0c419a33235d428ba328b43e6fe45a1e (127.0.97.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } }
I20260620 15:16:59.373032 7290 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } }
I20260620 15:16:59.373457 7290 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 15:16:59.373751 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca18fbf31a04a8ca99055a3367b1992. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } }
I20260620 15:16:59.373960 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 15:16:59.374255 7285 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: true } } }
I20260620 15:16:59.374449 7285 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 15:16:59.376120 7096 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 15:16:59.376549 7096 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 15:16:59.376710 7096 tablet_replica.cc:330] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: stopping tablet replica
I20260620 15:16:59.393886 7096 master.cc:552] Master@127.0.97.124:42865 shutdown complete.
I20260620 15:16:59.393965 7096 master_runner.cc:305] Clearing existing system tablet
I20260620 15:16:59.394340 7096 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:59.394429 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:59.397130 7096 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 15:16:59.400856 7292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:59.401298 7096 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:59.401557 7096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:16:59.401722 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal
uuid: "eb7bb01e759840caa7c9db33aaefe281"
format_stamp: "Formatted at 2026-06-20 15:16:59 on dist-test-slave-7s64"
server_key: "5446ef653f41e71050d2780ac55dfd9d"
server_key_iv: "dfa548c93ce41f663b17374cc61bc8b1"
server_key_version: "encryptionkey@0"
I20260620 15:16:59.406170 7096 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:16:59.412675 7096 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 15:16:59.412817 7096 log.cc:1196] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 15:16:59.413286 7096 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Deleting consensus metadata
I20260620 15:16:59.413537 7096 master_runner.cc:315] Copying system tablet from 127.0.97.126:46541
I20260620 15:16:59.415971 7096 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.126:46541
I20260620 15:16:59.431381 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.416706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:33334)
0620 15:16:59.417340 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.417363 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.417519 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.417899 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.417904 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.418419 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.418659 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.418670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.420082 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.420088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.421674 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.421688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.423039 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.423046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.423238 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.423936 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.423962 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.425047 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.428722 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.428727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.428731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.429245 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.429645 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.429650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.429654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.429814 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.430329 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.430334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.430517 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.430666 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.430966 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":365,"threads_started":1}
I20260620 15:16:59.431751 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.416739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.0.97.126:46541)
0620 15:16:59.416961 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.416968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.417474 (+ 506us) server_negotiation.cc:407] Connection header received
0620 15:16:59.417649 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.417654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.417752 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.417860 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.418829 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.419930 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.421823 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.422442 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.422559 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.425241 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.425247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.425252 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.425289 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.428599 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.429362 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.429367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.429371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.429519 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.429919 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.429923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.429927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.430215 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.430354 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.430755 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.430989 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 15:16:59.432277 6554 tablet_copy_service.cc:140] P bca18fbf31a04a8ca99055a3367b1992: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer eb7bb01e759840caa7c9db33aaefe281 ({username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:33334)
I20260620 15:16:59.432437 6554 tablet_copy_service.cc:161] P bca18fbf31a04a8ca99055a3367b1992: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer eb7bb01e759840caa7c9db33aaefe281 at {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:33334: session id = eb7bb01e759840caa7c9db33aaefe281-00000000000000000000000000000000
I20260620 15:16:59.436043 6554 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 15:16:59.438292 7096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 15:16:59.444962 7096 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: tablet copy: Starting download of 0 data blocks...
I20260620 15:16:59.445235 7096 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: tablet copy: Starting download of 1 WAL segments...
I20260620 15:16:59.447925 7096 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 15:16:59.452028 6554 tablet_copy_service.cc:342] P bca18fbf31a04a8ca99055a3367b1992: Request end of tablet copy session eb7bb01e759840caa7c9db33aaefe281-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:33334
I20260620 15:16:59.452159 6554 tablet_copy_service.cc:433] P bca18fbf31a04a8ca99055a3367b1992: ending tablet copy session eb7bb01e759840caa7c9db33aaefe281-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer eb7bb01e759840caa7c9db33aaefe281
I20260620 15:16:59.455719 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 6595
I20260620 15:16:59.462412 7096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:59.464963 7299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:59.469808 7300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:59.476621 7302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:59.478953 7096 server_base.cc:1034] running on GCE node
I20260620 15:16:59.479012 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:33143
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:46541
--webserver_port=35343
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.126:46541,127.0.97.125:37799
--tserver_master_addrs=127.0.97.126:46541,127.0.97.125:37799,127.0.97.124:42865 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:16:59.479353 7096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:59.479997 7096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:59.481248 7096 hybrid_clock.cc:648] HybridClock initialized: now 1781968619481217 us; error 26 us; skew 500 ppm
I20260620 15:16:59.481469 7096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:59.483240 7096 init.cc:377] Logged in from keytab as oryx/127.0.97.124@KRBTEST.COM (short username oryx)
I20260620 15:16:59.489565 7096 webserver.cc:460] Webserver started at http://127.0.97.124:36989/ using document root <none> and password file <none>
I20260620 15:16:59.489907 7096 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:59.490015 7096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:59.493290 7096 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:16:59.495355 7310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:59.495805 7096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 15:16:59.495971 7096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal
uuid: "eb7bb01e759840caa7c9db33aaefe281"
format_stamp: "Formatted at 2026-06-20 15:16:59 on dist-test-slave-7s64"
server_key: "5446ef653f41e71050d2780ac55dfd9d"
server_key_iv: "dfa548c93ce41f663b17374cc61bc8b1"
server_key_version: "encryptionkey@0"
I20260620 15:16:59.496160 7096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:59.527417 7096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:59.531306 7096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:59.555552 7361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:42865 every 8 connection(s)
I20260620 15:16:59.555732 7096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:42865
I20260620 15:16:59.556741 7096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
I20260620 15:16:59.566346 7362 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:16:59.570499 7362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Bootstrap starting.
I20260620 15:16:59.586321 7362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:16:59.586819 7362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Bootstrap complete.
I20260620 15:16:59.587791 7362 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } }
I20260620 15:16:59.588070 7362 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb7bb01e759840caa7c9db33aaefe281, State: Initialized, Role: LEARNER
I20260620 15:16:59.588306 7362 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } }
I20260620 15:16:59.589174 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.589636 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: This master's current role is: LEARNER
I20260620 15:16:59.589501 7362 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:16:59.601425 7376 catalog_manager.cc:1259] Loaded cluster ID: b14ee8ff09174c258de5f70796d4f926
I20260620 15:16:59.601563 7376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: loading cluster ID for follower catalog manager: success
I20260620 15:16:59.605140 7376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: acquiring CA information for follower catalog manager: success
I20260620 15:16:59.607470 7376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:16:59.804011 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.798201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42552 (local address 127.0.97.124:42865)
0620 15:16:59.798826 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.798831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.798849 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:16:59.798903 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.798906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.798970 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.799092 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:16:59.799484 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.800522 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.802158 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.803013 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.803153 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.803393 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.803557 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.803686 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":328,"threads_started":1}
I20260620 15:16:59.804011 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.798015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:42865 (local address 127.0.0.1:42552)
0620 15:16:59.798488 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.798512 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.798656 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.799113 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.799118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.799145 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:16:59.799339 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.799349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.800667 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.800672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.802024 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.802036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.803170 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.803176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.803363 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.803397 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.803577 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.803705 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":260,"threads_started":1}
I20260620 15:16:59.805028 7337 raft_consensus.cc:1203] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 15:16:59.805447 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bca18fbf31a04a8ca99055a3367b1992. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:16:59.805740 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: This master's current role is: LEARNER
W20260620 15:16:59.848958 7306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:16:59.849278 7306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:16:59.849457 7306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:16:59.857084 7306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:16:59.857223 7306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:16:59.857386 7306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:16:59.868588 7306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:33143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=35343
--tserver_master_addrs=127.0.97.126:46541,127.0.97.125:37799,127.0.97.124:42865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:16:59.870056 7306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:16:59.871654 7306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:16:59.887684 7386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:59.887751 7383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:16:59.887732 7384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:16:59.888759 7306 server_base.cc:1034] running on GCE node
I20260620 15:16:59.889321 7306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:16:59.890161 7306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:16:59.891342 7306 hybrid_clock.cc:648] HybridClock initialized: now 1781968619891293 us; error 52 us; skew 500 ppm
I20260620 15:16:59.891597 7306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:16:59.894665 7306 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260620 15:16:59.896035 7306 webserver.cc:460] Webserver started at http://127.0.97.65:35343/ using document root <none> and password file <none>
I20260620 15:16:59.896781 7306 fs_manager.cc:359] Metadata directory not provided
I20260620 15:16:59.896915 7306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:16:59.902400 7306 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 15:16:59.905078 7393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:59.906324 7306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:16:59.906503 7306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "22559783906942448d1eec02ed6331fe"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "09aa418c30fe6951ab0c4997239cc1a3"
server_key_iv: "cf3f61b274edeb21c9ec4eb9bba2a4b7"
server_key_version: "encryptionkey@0"
I20260620 15:16:59.907389 7306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:16:59.925233 7306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:16:59.929246 7306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:16:59.929505 7306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:16:59.930342 7306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:16:59.931977 7306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:16:59.932101 7306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:59.932206 7306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:16:59.932286 7306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:16:59.966257 7306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:33143
I20260620 15:16:59.966337 7505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:33143 every 8 connection(s)
I20260620 15:16:59.967619 7306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:16:59.974164 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 7306
I20260620 15:16:59.974467 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 6729
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968619, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
Jun 20 15:16:59 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.66@KRBTEST.COM
I20260620 15:16:59.996076 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.970344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:44745 (local address 127.0.97.125:37799)
0620 15:16:59.972857 (+ 2513us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.972864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.973281 (+ 417us) server_negotiation.cc:407] Connection header received
0620 15:16:59.976852 (+ 3571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.976861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.976963 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.977061 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.978520 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.979954 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.983307 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.984840 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.984992 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.989193 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.989200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.989204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.989244 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.992846 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.993657 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.993665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.993670 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.993813 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.994314 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.994319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.994323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.994670 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.994818 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.995437 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.995669 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2379,"thread_start_us":2279,"threads_started":1}
I20260620 15:16:59.996570 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.970304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:35769 (local address 127.0.97.126:46541)
0620 15:16:59.971399 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.971404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.972156 (+ 752us) server_negotiation.cc:407] Connection header received
0620 15:16:59.975283 (+ 3127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.975290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.975385 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.975533 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.977585 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.978312 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.980953 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.981587 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.981684 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.987312 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.987319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.987325 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.987365 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.991306 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.992919 (+ 1613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.992924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.992928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.993063 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.993630 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.993635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.993639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.994036 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.994171 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.995490 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.995691 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":845,"threads_started":1}
I20260620 15:16:59.996912 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.973046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:39107 (local address 127.0.97.124:42865)
0620 15:16:59.973206 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.973210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.973934 (+ 724us) server_negotiation.cc:407] Connection header received
0620 15:16:59.976716 (+ 2782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.976721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.976792 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.976885 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.979152 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.980203 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.982264 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.982859 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.982989 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.989892 (+ 6903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.989903 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.989910 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.989955 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:16:59.993850 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.994691 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.994696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.994699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.994879 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:16:59.995363 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:16:59.995369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:16:59.995372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:16:59.995663 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:16:59.995782 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:16:59.996279 (+ 497us) server_negotiation.cc:299] Negotiation successful
0620 15:16:59.996541 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:16:59.998260 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.970938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.65:35769)
0620 15:16:59.971994 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.972054 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.974981 (+ 2927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.975919 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.975927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.976539 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.977372 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.977414 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.978461 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.978466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.980778 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.980795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.982018 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.982025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.982357 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.983086 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.983116 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.987088 (+ 3972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.991526 (+ 4438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.991542 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.991558 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.992749 (+ 1191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.993211 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.993216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.993220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.993443 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.994264 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.994272 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.994718 (+ 446us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.997344 (+ 2626us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.997644 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":449,"spinlock_wait_cycles":34048,"thread_start_us":1129,"threads_started":1}
I20260620 15:16:59.998299 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.972922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:42865 (local address 127.0.97.65:39107)
0620 15:16:59.973832 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.973846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.976559 (+ 2713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.977043 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.977047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.978668 (+ 1621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.978997 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.979008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.980445 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.980450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.982117 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.982128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.983266 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.983274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.983451 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.984090 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.984112 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.989624 (+ 5512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.994028 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.994032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.994035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.994563 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.995083 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.995087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.995091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.995231 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.995857 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.995861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.995989 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.997074 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.997333 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":13,"spinlock_wait_cycles":5504,"thread_start_us":653,"threads_started":1}
I20260620 15:16:59.998344 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.970938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:37799 (local address 127.0.97.65:44745)
0620 15:16:59.971994 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0620 15:16:59.972049 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:16:59.974981 (+ 2932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:16:59.977370 (+ 2389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:16:59.977380 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:16:59.977985 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:16:59.978300 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.978314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.980198 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.980206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.983145 (+ 2939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:16:59.983157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.985056 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.985069 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:16:59.985374 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.986026 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:16:59.986043 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:16:59.988851 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:16:59.993006 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.993012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.993015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.993515 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.993961 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:16:59.993967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:16:59.993971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:16:59.994174 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:16:59.994904 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:16:59.994909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:16:59.995037 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 15:16:59.996273 (+ 1236us) client_negotiation.cc:240] Negotiation successful
0620 15:16:59.996810 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":385,"spinlock_wait_cycles":2432,"thread_start_us":465,"threads_started":1}
I20260620 15:16:59.999959 7507 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46541
I20260620 15:17:00.000058 7506 heartbeater.cc:347] Connected to a master server at 127.0.97.124:42865
I20260620 15:17:00.000092 7508 heartbeater.cc:347] Connected to a master server at 127.0.97.125:37799
I20260620 15:17:00.000286 7506 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:00.000289 7508 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:00.000286 7507 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:00.001421 7507 heartbeater.cc:511] Master 127.0.97.126:46541 requested a full tablet report, sending...
I20260620 15:17:00.001456 7506 heartbeater.cc:511] Master 127.0.97.124:42865 requested a full tablet report, sending...
I20260620 15:17:00.001484 7508 heartbeater.cc:511] Master 127.0.97.125:37799 requested a full tablet report, sending...
I20260620 15:17:00.003676 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:16:59.975285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.0.97.66:37201)
0620 15:16:59.982610 (+ 7325us) server_negotiation.cc:206] Beginning negotiation
0620 15:16:59.982620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:16:59.982653 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:16:59.982718 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:16:59.982726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:16:59.982940 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:16:59.983772 (+ 832us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:16:59.985599 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.987095 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.989001 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:16:59.989828 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:16:59.989947 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:16:59.996560 (+ 6613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:16:59.996572 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:16:59.996586 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:16:59.996621 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:00.000519 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:00.001390 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:00.001400 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:00.001405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:00.001555 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:00.001984 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:00.001989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:00.001995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:00.002403 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:00.002697 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:00.003026 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 15:17:00.003274 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7182,"thread_start_us":7031,"threads_started":1}
I20260620 15:17:00.004220 7327 ts_manager.cc:194] Registered new tserver with Master: 22559783906942448d1eec02ed6331fe (127.0.97.65:33143)
I20260620 15:17:00.004701 7132 ts_manager.cc:194] Registered new tserver with Master: 22559783906942448d1eec02ed6331fe (127.0.97.65:33143)
I20260620 15:17:00.005476 6534 ts_manager.cc:194] Re-registered known tserver with Master: 22559783906942448d1eec02ed6331fe (127.0.97.65:33143)
I20260620 15:17:00.005914 6852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:00.006755 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:35769
I20260620 15:17:00.089279 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 6729
I20260620 15:17:00.108637 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:37201
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:46541
--webserver_port=43729
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.126:46541,127.0.97.125:37799
--tserver_master_addrs=127.0.97.126:46541,127.0.97.125:37799,127.0.97.124:42865 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:17:00.340238 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: true } } }
I20260620 15:17:00.340696 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:00.342239 7520 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992: attempting to promote NON_VOTER eb7bb01e759840caa7c9db33aaefe281 to VOTER
I20260620 15:17:00.342991 7520 consensus_queue.cc:237] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } }
I20260620 15:17:00.345183 7142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Refusing update from remote peer bca18fbf31a04a8ca99055a3367b1992: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 15:17:00.345788 7337 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 LEARNER]: Refusing update from remote peer bca18fbf31a04a8ca99055a3367b1992: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 15:17:00.345826 7519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 15:17:00.346374 7278 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 15:17:00.348920 7520 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eb7bb01e759840caa7c9db33aaefe281 (127.0.97.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:00.349735 7142 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eb7bb01e759840caa7c9db33aaefe281 (127.0.97.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:00.349843 7337 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, eb7bb01e759840caa7c9db33aaefe281 (127.0.97.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:00.353009 7523 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:00.353330 7523 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:17:00.355892 7520 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:00.356174 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:00.356549 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:00.356808 7278 sys_catalog.cc:455] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:00.357169 7278 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:00.356197 7520 sys_catalog.cc:458] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992 [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:00.361231 7524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bca18fbf31a04a8ca99055a3367b1992" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:00.361604 7524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: This master's current role is: FOLLOWER
W20260620 15:17:00.432715 7518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:00.433014 7518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:00.433178 7518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:00.440223 7518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:00.440364 7518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:00.440549 7518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:17:00.449681 7518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:37201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=43729
--tserver_master_addrs=127.0.97.126:46541,127.0.97.125:37799,127.0.97.124:42865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:00.450876 7518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:00.452090 7518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:00.465032 7533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:00.465114 7530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:00.465093 7531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:00.466017 7518 server_base.cc:1034] running on GCE node
I20260620 15:17:00.466606 7518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:00.467391 7518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:00.468592 7518 hybrid_clock.cc:648] HybridClock initialized: now 1781968620468552 us; error 30 us; skew 500 ppm
I20260620 15:17:00.468897 7518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:00 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968620, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:00.471793 7518 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260620 15:17:00.473214 7518 webserver.cc:460] Webserver started at http://127.0.97.66:43729/ using document root <none> and password file <none>
I20260620 15:17:00.473841 7518 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:00.473994 7518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:00.478996 7518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:17:00.481596 7540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:00.482758 7518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:17:00.482931 7518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "4bf8ff273da161a0306aeacc7ba3eee8"
server_key_iv: "9047c4c17b30091f2e826b1b144db632"
server_key_version: "encryptionkey@0"
I20260620 15:17:00.483834 7518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:00.496341 7518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:00.501173 7518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:00.501466 7518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:00.502347 7518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:00.503739 7518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:00.503852 7518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:00.503960 7518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:00.504040 7518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:00.535539 7518 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37201
I20260620 15:17:00.535627 7652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37201 every 8 connection(s)
I20260620 15:17:00.537276 7518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:17:00.546949 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 7518
I20260620 15:17:00.547252 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 6862
Jun 20 15:17:00 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.67@KRBTEST.COM
I20260620 15:17:00.571921 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:00.548954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.0.97.67:39053)
0620 15:17:00.549665 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:00.549675 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:00.549704 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:17:00.549777 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:00.549784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:00.550046 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:00.550199 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:00.553672 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.555485 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.559171 (+ 3686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.559854 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.560040 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:00.566084 (+ 6044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:00.566093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:00.566104 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:00.566136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:00.569144 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:00.569790 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:00.569796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:00.569799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:00.569929 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:00.570304 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:00.570313 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:00.570315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:00.570629 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:00.570926 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:00.571236 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 15:17:00.571492 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":410,"threads_started":1}
I20260620 15:17:00.573150 6985 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 20 15:17:00 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968620, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
Jun 20 15:17:00 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968620, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
Jun 20 15:17:00 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968620, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260620 15:17:00.593914 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:00.539931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:54323 (local address 127.0.97.124:42865)
0620 15:17:00.560943 (+ 21012us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:00.560951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:00.560974 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:17:00.574254 (+ 13280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:00.574261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:00.574360 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:00.574429 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:00.576765 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.577960 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.580864 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.581531 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.581613 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:00.586557 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:00.586567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:00.586573 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:00.586610 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:00.590230 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:00.591144 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:00.591149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:00.591153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:00.591288 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:00.591850 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:00.591855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:00.591858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:00.592191 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:00.592314 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:00.593248 (+ 934us) server_negotiation.cc:299] Negotiation successful
0620 15:17:00.593491 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20820,"thread_start_us":29304,"threads_started":1}
I20260620 15:17:00.595204 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:00.539980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:33769 (local address 127.0.97.125:37799)
0620 15:17:00.560396 (+ 20416us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:00.560402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:00.560421 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:17:00.577115 (+ 16694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:00.577121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:00.577211 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:00.577286 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:00.578856 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.580042 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.582268 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.583515 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.583626 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:00.588222 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:00.588229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:00.588233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:00.588277 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:00.592061 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:00.592851 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:00.592857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:00.592861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:00.593005 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:00.593444 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:00.593449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:00.593453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:00.593816 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:00.593963 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:00.594536 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 15:17:00.594748 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20289,"thread_start_us":20435,"threads_started":1}
I20260620 15:17:00.595535 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:00.553138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.66:52943)
0620 15:17:00.557176 (+ 4038us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:00.557193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:00.573749 (+ 16556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:00.576828 (+ 3079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:00.576835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:00.577396 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:00.577721 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:00.577737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.579628 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.579634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:00.581217 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:00.581230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.582230 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.582236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:00.582383 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:00.583091 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:00.583131 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:00.586705 (+ 3574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:00.591447 (+ 4742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:00.591451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:00.591455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:00.591982 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:00.592391 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:00.592395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:00.592398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:00.592578 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:00.593179 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:00.593185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:00.593311 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 15:17:00.593465 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 15:17:00.594015 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3913,"mutex_wait_us":26,"thread_start_us":3776,"threads_started":1}
I20260620 15:17:00.595937 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:00.541345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:42865 (local address 127.0.97.66:54323)
0620 15:17:00.552920 (+ 11575us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:00.552943 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:00.574128 (+ 21185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:00.574979 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:00.574991 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:00.575781 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:00.576607 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:00.576627 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.578528 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.578532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:00.580700 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:00.580713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.581945 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.581952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:00.582242 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:00.583163 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:00.583187 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:00.586308 (+ 3121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:00.590378 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:00.590386 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:00.590406 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:00.591007 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:00.591444 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:00.591449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:00.591452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:00.591714 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:00.592398 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:00.592410 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:00.592840 (+ 430us) client_negotiation.cc:769] Sending connection context
0620 15:17:00.594152 (+ 1312us) client_negotiation.cc:240] Negotiation successful
0620 15:17:00.594396 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11433,"mutex_wait_us":6,"spinlock_wait_cycles":23296,"thread_start_us":11275,"threads_started":1}
I20260620 15:17:00.596244 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:00.553253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52943 (local address 127.0.97.126:46541)
0620 15:17:00.573335 (+ 20082us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:00.573341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:00.573360 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:17:00.576606 (+ 3246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:00.576613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:00.576699 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:00.576810 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:00.577894 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.579451 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.581398 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.582041 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.582642 (+ 601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:00.586905 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:00.586911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:00.586915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:00.586972 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:00.591329 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:00.592123 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:00.592128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:00.592131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:00.592267 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:00.592720 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:00.592725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:00.592728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:00.593038 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:00.595515 (+ 2477us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:00.595665 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 15:17:00.595880 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19929,"thread_start_us":20101,"threads_started":1}
I20260620 15:17:00.596328 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:00.540756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:37799 (local address 127.0.97.66:33769)
0620 15:17:00.549939 (+ 9183us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:00.549985 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:00.576944 (+ 26959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:00.577533 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:00.577538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:00.578001 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:00.578622 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:00.578636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.580235 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.580241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:00.582081 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:00.582097 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:00.583978 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:00.583985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:00.584230 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:00.584962 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:00.584989 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:00.588032 (+ 3043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:00.592206 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:00.592212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:00.592215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:00.592725 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:00.593144 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:00.593149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:00.593153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:00.593319 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:00.594052 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:00.594059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:00.594226 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 15:17:00.595464 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0620 15:17:00.595754 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8881,"mutex_wait_us":46,"thread_start_us":8685,"threads_started":1}
I20260620 15:17:00.597473 7655 heartbeater.cc:347] Connected to a master server at 127.0.97.125:37799
I20260620 15:17:00.597573 7654 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46541
I20260620 15:17:00.597807 7654 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:00.597867 7653 heartbeater.cc:347] Connected to a master server at 127.0.97.124:42865
I20260620 15:17:00.597988 7653 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:00.597811 7655 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:00.598817 7654 heartbeater.cc:511] Master 127.0.97.126:46541 requested a full tablet report, sending...
I20260620 15:17:00.599310 7655 heartbeater.cc:511] Master 127.0.97.125:37799 requested a full tablet report, sending...
I20260620 15:17:00.600405 6534 ts_manager.cc:194] Re-registered known tserver with Master: d3c3f0c9d7ff4262a9fab6dd8d53204b (127.0.97.66:37201)
I20260620 15:17:00.600682 7132 ts_manager.cc:194] Registered new tserver with Master: d3c3f0c9d7ff4262a9fab6dd8d53204b (127.0.97.66:37201)
I20260620 15:17:00.601460 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:52943
I20260620 15:17:00.601867 7653 heartbeater.cc:511] Master 127.0.97.124:42865 requested a full tablet report, sending...
I20260620 15:17:00.602857 7327 ts_manager.cc:194] Registered new tserver with Master: d3c3f0c9d7ff4262a9fab6dd8d53204b (127.0.97.66:37201)
I20260620 15:17:00.665282 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 6862
I20260620 15:17:00.680825 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:39053
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:46541
--webserver_port=33081
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.126:46541,127.0.97.125:37799
--tserver_master_addrs=127.0.97.126:46541,127.0.97.125:37799,127.0.97.124:42865 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:01.000048 7666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:01.000303 7666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:01.000455 7666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:01.008713 7666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:01.008810 7666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:01.008924 7666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:17:01.009372 7507 heartbeater.cc:503] Master 127.0.97.126:46541 was elected leader, sending a full tablet report...
I20260620 15:17:01.018944 7666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:39053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=33081
--tserver_master_addrs=127.0.97.126:46541,127.0.97.125:37799,127.0.97.124:42865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:01.020170 7666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:01.021459 7666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:01.034269 7674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:01.034328 7672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:01.034358 7671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:01.037911 7666 server_base.cc:1034] running on GCE node
I20260620 15:17:01.038434 7666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:01.039255 7666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:01.040442 7666 hybrid_clock.cc:648] HybridClock initialized: now 1781968621040383 us; error 56 us; skew 500 ppm
I20260620 15:17:01.040807 7666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:01 dist-test-slave-7s64 krb5kdc[6489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:01.043818 7666 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260620 15:17:01.045346 7666 webserver.cc:460] Webserver started at http://127.0.97.67:33081/ using document root <none> and password file <none>
I20260620 15:17:01.046033 7666 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:01.046142 7666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:01.051405 7666 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:17:01.053951 7681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:01.055243 7666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:17:01.055402 7666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "fe55b5749b6843c899d0d2d8896c31cf"
format_stamp: "Formatted at 2026-06-20 15:16:57 on dist-test-slave-7s64"
server_key: "eb168d831ce778cd1bc44057811d63ce"
server_key_iv: "fd26d9497e380ff3405c22c63df0910f"
server_key_version: "encryptionkey@0"
I20260620 15:17:01.056298 7666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:01.069243 7666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:01.073173 7666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:01.073436 7666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:01.074335 7666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:01.075583 7666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:01.075717 7666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:01.075838 7666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:01.075924 7666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:01.109200 7666 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39053
I20260620 15:17:01.109282 7793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39053 every 8 connection(s)
I20260620 15:17:01.110680 7666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:17:01.119196 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 7666
Jun 20 15:17:01 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
Jun 20 15:17:01 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
Jun 20 15:17:01 dist-test-slave-7s64 krb5kdc[6489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968621, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
I20260620 15:17:01.142405 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.113430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:38441 (local address 127.0.97.126:46541)
0620 15:17:01.113901 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.113907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.119586 (+ 5679us) server_negotiation.cc:407] Connection header received
0620 15:17:01.119687 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.119695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.119796 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.119932 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:01.122568 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.124017 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.127656 (+ 3639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.128307 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.128444 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.133242 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:01.133248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:01.133255 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:01.133304 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:01.137127 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.138704 (+ 1577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.138710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.138714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.138850 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.139579 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.139584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.139596 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.139941 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:01.140073 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.141860 (+ 1787us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.142062 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":254,"threads_started":1}
I20260620 15:17:01.143363 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.113956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.67:38441)
0620 15:17:01.115050 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:01.115103 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:01.118801 (+ 3698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:01.120405 (+ 1604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:01.120415 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:01.121381 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:01.122333 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.122361 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.124256 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.124265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.127101 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.127121 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.128542 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.128551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.128780 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.129478 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:01.129498 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:01.133018 (+ 3520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:01.138015 (+ 4997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:01.138027 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:01.138044 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:01.138574 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:01.139322 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:01.139324 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:01.139327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:01.139471 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:01.140866 (+ 1395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:01.140882 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:01.141271 (+ 389us) client_negotiation.cc:769] Sending connection context
0620 15:17:01.141551 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 15:17:01.142050 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":492,"threads_started":1}
I20260620 15:17:01.143553 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.114218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:39145 (local address 127.0.97.125:37799)
0620 15:17:01.119482 (+ 5264us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.119491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.119516 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:17:01.125001 (+ 5485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.125010 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.125129 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.125226 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:01.128307 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.129451 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.131316 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.131928 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.132042 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.136449 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:01.136458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:01.136464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:01.136566 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:01.139813 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.140725 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.140732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.140736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.140879 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.141268 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.141273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.141276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.141600 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:01.141721 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.142999 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.143194 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5119,"thread_start_us":5015,"threads_started":1}
I20260620 15:17:01.144429 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.113959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:37799 (local address 127.0.97.67:39145)
0620 15:17:01.116471 (+ 2512us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:01.116517 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:01.124659 (+ 8142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:01.125504 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:01.125511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:01.126294 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:01.126701 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.126730 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.129605 (+ 2875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.129612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.131163 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.131175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.132140 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.132150 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.132403 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.133096 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:01.133122 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:01.136202 (+ 3080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:01.139999 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:01.140003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:01.140007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:01.140476 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:01.141002 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:01.141006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:01.141009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:01.141141 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:01.142615 (+ 1474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:01.142619 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:01.142752 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 15:17:01.143761 (+ 1009us) client_negotiation.cc:240] Negotiation successful
0620 15:17:01.144023 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2341,"mutex_wait_us":50,"spinlock_wait_cycles":35712,"thread_start_us":13643,"threads_started":1}
I20260620 15:17:01.147881 7795 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46541
I20260620 15:17:01.147902 7796 heartbeater.cc:347] Connected to a master server at 127.0.97.125:37799
I20260620 15:17:01.148237 7796 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:01.149144 7796 heartbeater.cc:511] Master 127.0.97.125:37799 requested a full tablet report, sending...
I20260620 15:17:01.148234 7795 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:01.150159 7795 heartbeater.cc:511] Master 127.0.97.126:46541 requested a full tablet report, sending...
I20260620 15:17:01.150930 7132 ts_manager.cc:194] Registered new tserver with Master: fe55b5749b6843c899d0d2d8896c31cf (127.0.97.67:39053)
I20260620 15:17:01.151027 6534 ts_manager.cc:194] Re-registered known tserver with Master: fe55b5749b6843c899d0d2d8896c31cf (127.0.97.67:39053)
I20260620 15:17:01.152103 6534 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:38441
I20260620 15:17:01.155866 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.115783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:42865 (local address 127.0.97.67:44305)
0620 15:17:01.128017 (+ 12234us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:01.128034 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:01.128145 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:01.128584 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:01.128589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:01.129152 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:01.129436 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.129446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.133232 (+ 3786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.133237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.134904 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.134919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.136033 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.136039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.136240 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.136757 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:01.136783 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:01.139226 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:01.153402 (+ 14176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:01.153407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:01.153412 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:01.153932 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:01.154346 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:01.154351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:01.154354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:01.154521 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:01.155188 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:01.155191 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:01.155302 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 15:17:01.155400 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:17:01.155603 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12122,"thread_start_us":320,"threads_started":1}
I20260620 15:17:01.156109 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.115847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:44305 (local address 127.0.97.124:42865)
0620 15:17:01.127548 (+ 11701us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.127553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.128132 (+ 579us) server_negotiation.cc:407] Connection header received
0620 15:17:01.128265 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.128273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.128361 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.128470 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:01.131022 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.132074 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.135097 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.135817 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.136552 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.139729 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:01.139735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:01.139740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:01.139770 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:01.153287 (+ 13517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.154069 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.154075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.154080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.154218 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.154659 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.154666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.154672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.155054 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:01.155198 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.155547 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.155766 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11600,"thread_start_us":11515,"threads_started":1}
I20260620 15:17:01.157349 7794 heartbeater.cc:347] Connected to a master server at 127.0.97.124:42865
I20260620 15:17:01.157485 7794 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:01.157900 7794 heartbeater.cc:511] Master 127.0.97.124:42865 requested a full tablet report, sending...
I20260620 15:17:01.159228 7327 ts_manager.cc:194] Registered new tserver with Master: fe55b5749b6843c899d0d2d8896c31cf (127.0.97.67:39053)
I20260620 15:17:01.172120 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.151129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.0.97.126:46541)
0620 15:17:01.151293 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.151300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.152650 (+ 1350us) server_negotiation.cc:407] Connection header received
0620 15:17:01.152809 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.152816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.152907 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.153033 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:01.155106 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.156140 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.159538 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.160444 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.160594 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.164298 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:01.164305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:01.164309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:01.164344 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:01.168019 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.168961 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.168968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.168972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.169110 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.169568 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.169574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.169579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.169947 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:01.170159 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.171185 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.171378 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:17:01.177074 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.157306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.0.97.124:42865)
0620 15:17:01.157515 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.157521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.161149 (+ 3628us) server_negotiation.cc:407] Connection header received
0620 15:17:01.161281 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.161284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.161332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.161433 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:01.163273 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.163993 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.166003 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.166651 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.166772 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.170334 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:01.170340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:01.170344 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:01.170380 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:01.174085 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.174916 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.174921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.174925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.175050 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.175595 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.175600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.175605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.175933 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:01.176213 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.176534 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.176737 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 15:17:01.177394 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.151528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.0.97.125:37799)
0620 15:17:01.151677 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.151682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.153906 (+ 2224us) server_negotiation.cc:407] Connection header received
0620 15:17:01.154144 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.154151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.154226 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.154290 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:01.162947 (+ 8657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.164106 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.165820 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.166401 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.166549 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.170113 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:01.170117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:01.170120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:01.170148 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:01.173773 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.175286 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.175291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.175294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.175441 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.175799 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.175803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.175807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.176124 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:01.176376 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.176924 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.177088 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:17:01.176182 6534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:17:01.183836 6534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:17:01.223073 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.199931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37201 (local address 127.0.0.1:52642)
0620 15:17:01.214632 (+ 14701us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:01.214653 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:01.214768 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:01.215716 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:01.215723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:01.215757 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:01.216243 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.216256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.218178 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.218183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.220379 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.220390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.222128 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.222134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.222330 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.222371 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:17:01.222478 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 15:17:01.222613 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14582,"mutex_wait_us":6,"thread_start_us":14931,"threads_started":1}
I20260620 15:17:01.223295 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.199896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.0.97.67:39053)
0620 15:17:01.212807 (+ 12911us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.212818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.212857 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:17:01.212983 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.212995 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.213303 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.213572 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:01.214147 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.216015 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.219015 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.221008 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.221162 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.221424 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.222874 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.222998 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12704,"thread_start_us":13951,"threads_started":1}
I20260620 15:17:01.223340 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.199667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39053 (local address 127.0.0.1:35194)
0620 15:17:01.206522 (+ 6855us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:01.206543 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:01.206656 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:01.213557 (+ 6901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:01.213564 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:01.213625 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:01.213956 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.213970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.216537 (+ 2567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.216542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.218691 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.218706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.222469 (+ 3763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.222475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.222636 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.222705 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 15:17:01.222797 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 15:17:01.222912 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6731,"mutex_wait_us":24,"thread_start_us":6579,"threads_started":1}
I20260620 15:17:01.223778 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.200198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.0.97.66:37201)
0620 15:17:01.208941 (+ 8743us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.208949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.214746 (+ 5797us) server_negotiation.cc:407] Connection header received
0620 15:17:01.215025 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.215031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.215295 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.215476 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:01.216475 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.218005 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.220539 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.221957 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.222104 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.222315 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.223360 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.223539 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8574,"thread_start_us":8475,"threads_started":1}
I20260620 15:17:01.226917 7729 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea36abb3b4243da95478ffbb566e45f (DEFAULT_TABLE table=test-table [id=7b4ac1716948432b83d161d3e04a1517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:01.226909 7588 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea36abb3b4243da95478ffbb566e45f (DEFAULT_TABLE table=test-table [id=7b4ac1716948432b83d161d3e04a1517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:01.228258 7729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea36abb3b4243da95478ffbb566e45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:01.228304 7588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea36abb3b4243da95478ffbb566e45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:01.231029 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.215421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.0.97.65:33143)
0620 15:17:01.216673 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.216684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.221726 (+ 5042us) server_negotiation.cc:407] Connection header received
0620 15:17:01.221897 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.221904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.222096 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.222274 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:01.224362 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.225805 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.228651 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.229965 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.230110 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.230293 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.230560 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.230723 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":860,"threads_started":1}
I20260620 15:17:01.231630 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.215144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:33143 (local address 127.0.0.1:38328)
0620 15:17:01.221603 (+ 6459us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:01.221616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:01.221765 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:01.222263 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:01.222268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:01.222290 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:01.222689 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.222697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.225945 (+ 3248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.225951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.227871 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.227887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.230181 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.230187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.230367 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.230410 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 15:17:01.230507 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 15:17:01.230622 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6338,"mutex_wait_us":1,"thread_start_us":6710,"threads_started":1}
I20260620 15:17:01.234935 7441 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea36abb3b4243da95478ffbb566e45f (DEFAULT_TABLE table=test-table [id=7b4ac1716948432b83d161d3e04a1517]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:01.236171 7441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ea36abb3b4243da95478ffbb566e45f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:01.241043 7825 tablet_bootstrap.cc:492] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: Bootstrap starting.
I20260620 15:17:01.243002 7826 tablet_bootstrap.cc:492] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: Bootstrap starting.
I20260620 15:17:01.244907 7825 tablet_bootstrap.cc:654] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:01.246212 7825 log.cc:824] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:01.246395 7826 tablet_bootstrap.cc:654] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:01.247880 7826 log.cc:824] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:01.249316 7825 tablet_bootstrap.cc:492] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: No bootstrap required, opened a new log
I20260620 15:17:01.249584 7825 ts_tablet_manager.cc:1397] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 15:17:01.250738 7826 tablet_bootstrap.cc:492] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: No bootstrap required, opened a new log
I20260620 15:17:01.250981 7826 ts_tablet_manager.cc:1397] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:17:01.250866 7829 tablet_bootstrap.cc:492] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: Bootstrap starting.
I20260620 15:17:01.253860 7829 tablet_bootstrap.cc:654] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:01.255577 7829 log.cc:824] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:01.257863 7829 tablet_bootstrap.cc:492] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: No bootstrap required, opened a new log
I20260620 15:17:01.258114 7829 ts_tablet_manager.cc:1397] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 15:17:01.260558 7825 raft_consensus.cc:348] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } }
I20260620 15:17:01.260952 7825 raft_consensus.cc:374] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:01.261047 7825 raft_consensus.cc:729] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c3f0c9d7ff4262a9fab6dd8d53204b, State: Initialized, Role: FOLLOWER
I20260620 15:17:01.261544 7826 raft_consensus.cc:348] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } }
I20260620 15:17:01.262061 7825 consensus_queue.cc:260] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } }
I20260620 15:17:01.262343 7826 raft_consensus.cc:374] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:01.262437 7826 raft_consensus.cc:729] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe55b5749b6843c899d0d2d8896c31cf, State: Initialized, Role: FOLLOWER
I20260620 15:17:01.263207 7826 consensus_queue.cc:260] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } }
I20260620 15:17:01.265273 7826 ts_tablet_manager.cc:1428] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 15:17:01.265569 7795 heartbeater.cc:503] Master 127.0.97.126:46541 was elected leader, sending a full tablet report...
I20260620 15:17:01.267098 7654 heartbeater.cc:503] Master 127.0.97.126:46541 was elected leader, sending a full tablet report...
I20260620 15:17:01.270004 7825 ts_tablet_manager.cc:1428] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: Time spent starting tablet: real 0.020s user 0.011s sys 0.006s
I20260620 15:17:01.271198 7829 raft_consensus.cc:348] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } }
I20260620 15:17:01.272681 7829 raft_consensus.cc:374] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:01.272784 7829 raft_consensus.cc:729] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22559783906942448d1eec02ed6331fe, State: Initialized, Role: FOLLOWER
I20260620 15:17:01.273290 7829 consensus_queue.cc:260] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } }
I20260620 15:17:01.274847 7829 ts_tablet_manager.cc:1428] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
W20260620 15:17:01.308641 7656 tablet.cc:2369] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:17:01.364836 7797 tablet.cc:2369] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:17:01.471592 7509 tablet.cc:2369] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:17:01.549609 7831 raft_consensus.cc:482] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:17:01.549937 7831 raft_consensus.cc:504] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } }
I20260620 15:17:01.551862 7831 leader_election.cc:288] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe55b5749b6843c899d0d2d8896c31cf (127.0.97.67:39053), 22559783906942448d1eec02ed6331fe (127.0.97.65:33143)
I20260620 15:17:01.558538 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.552353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:33035 (local address 127.0.97.65:33143)
0620 15:17:01.552739 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.552748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.553066 (+ 318us) server_negotiation.cc:407] Connection header received
0620 15:17:01.553310 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.553314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.553412 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.553513 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:01.554039 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.554744 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.556509 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.557638 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.557774 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.557871 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.558174 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.558301 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260620 15:17:01.558568 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.552226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:33143 (local address 127.0.97.66:33035)
0620 15:17:01.552961 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:01.552976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:01.553147 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:01.553535 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:01.553539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:01.553560 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:01.553897 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.553905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.554911 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.554916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.556344 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.556355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.557844 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.557850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.558013 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.558045 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:17:01.558134 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 15:17:01.558252 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":372,"threads_started":1}
I20260620 15:17:01.559451 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea36abb3b4243da95478ffbb566e45f" candidate_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22559783906942448d1eec02ed6331fe" is_pre_election: true
I20260620 15:17:01.559645 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.552264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:55445 (local address 127.0.97.67:39053)
0620 15:17:01.552514 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.552522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.553379 (+ 857us) server_negotiation.cc:407] Connection header received
0620 15:17:01.553565 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.553569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.553626 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.553725 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:01.554201 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.554919 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.557233 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.558311 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.558435 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.558524 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.559268 (+ 744us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.559399 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:17:01.559976 7461 raft_consensus.cc:2454] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3c3f0c9d7ff4262a9fab6dd8d53204b in term 0.
I20260620 15:17:01.560094 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.552303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39053 (local address 127.0.97.66:55445)
0620 15:17:01.553263 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:01.553283 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:01.553423 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:01.553741 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:01.553745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:01.553774 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:01.554064 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.554072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.555054 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.555060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.557089 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:01.557101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.558977 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.558984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:01.559112 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.559134 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:17:01.559235 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:17:01.559357 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":699,"threads_started":1}
I20260620 15:17:01.560700 7541 leader_election.cc:302] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22559783906942448d1eec02ed6331fe, d3c3f0c9d7ff4262a9fab6dd8d53204b; no voters:
I20260620 15:17:01.561120 7831 raft_consensus.cc:2781] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:17:01.561246 7831 raft_consensus.cc:482] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:17:01.561324 7831 raft_consensus.cc:3037] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:01.561553 7749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea36abb3b4243da95478ffbb566e45f" candidate_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe55b5749b6843c899d0d2d8896c31cf" is_pre_election: true
I20260620 15:17:01.562062 7749 raft_consensus.cc:2454] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3c3f0c9d7ff4262a9fab6dd8d53204b in term 0.
I20260620 15:17:01.564419 7831 raft_consensus.cc:504] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } }
I20260620 15:17:01.565048 7831 leader_election.cc:288] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [CANDIDATE]: Term 1 election: Requested vote from peers fe55b5749b6843c899d0d2d8896c31cf (127.0.97.67:39053), 22559783906942448d1eec02ed6331fe (127.0.97.65:33143)
I20260620 15:17:01.565480 7749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea36abb3b4243da95478ffbb566e45f" candidate_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe55b5749b6843c899d0d2d8896c31cf"
I20260620 15:17:01.565627 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ea36abb3b4243da95478ffbb566e45f" candidate_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22559783906942448d1eec02ed6331fe"
I20260620 15:17:01.565675 7749 raft_consensus.cc:3037] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:01.565828 7461 raft_consensus.cc:3037] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:01.569315 7461 raft_consensus.cc:2454] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3c3f0c9d7ff4262a9fab6dd8d53204b in term 1.
I20260620 15:17:01.569316 7749 raft_consensus.cc:2454] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3c3f0c9d7ff4262a9fab6dd8d53204b in term 1.
I20260620 15:17:01.569684 7541 leader_election.cc:302] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22559783906942448d1eec02ed6331fe, d3c3f0c9d7ff4262a9fab6dd8d53204b; no voters:
I20260620 15:17:01.569941 7831 raft_consensus.cc:2781] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:01.570459 7831 raft_consensus.cc:686] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 1 LEADER]: Becoming Leader. State: Replica: d3c3f0c9d7ff4262a9fab6dd8d53204b, State: Running, Role: LEADER
I20260620 15:17:01.570922 7831 consensus_queue.cc:237] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } }
I20260620 15:17:01.576056 6534 catalog_manager.cc:5495] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b reported cstate change: term changed from 0 to 1, leader changed from <none> to d3c3f0c9d7ff4262a9fab6dd8d53204b (127.0.97.66). New cstate: current_term: 1 leader_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } health_report { overall_health: UNKNOWN } } }
I20260620 15:17:01.670264 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.665733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.0.97.66:37201)
0620 15:17:01.665909 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.665913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.665930 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:17:01.666100 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.666103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.666175 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.666294 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:17:01.666589 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.667236 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.668450 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.668889 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.668984 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.669356 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:17:01.669843 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:17:01.669955 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.670019 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.670077 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:17:01.678122 7749 raft_consensus.cc:1261] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf [term 1 FOLLOWER]: Refusing update from remote peer d3c3f0c9d7ff4262a9fab6dd8d53204b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:01.678122 7461 raft_consensus.cc:1261] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe [term 1 FOLLOWER]: Refusing update from remote peer d3c3f0c9d7ff4262a9fab6dd8d53204b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:01.679066 7836 consensus_queue.cc:1035] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:01.679396 7831 consensus_queue.cc:1035] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [LEADER]: Connected to new peer: Peer: permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:01.691103 7841 mvcc.cc:205] Tried to move back new op lower bound from 7298943474383630336 to 7298943473958440960. Current Snapshot: MvccSnapshot[applied={T|T < 7298943474383630336}]
I20260620 15:17:01.692152 7840 mvcc.cc:205] Tried to move back new op lower bound from 7298943474383630336 to 7298943473958440960. Current Snapshot: MvccSnapshot[applied={T|T < 7298943474383630336}]
I20260620 15:17:01.705895 6533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:01.706171 6533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:01.713304 6533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bca18fbf31a04a8ca99055a3367b1992: Sending DeleteTablet for 3 replicas of tablet 6ea36abb3b4243da95478ffbb566e45f
I20260620 15:17:01.714493 7441 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ea36abb3b4243da95478ffbb566e45f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:01 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:38328
I20260620 15:17:01.714574 7729 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ea36abb3b4243da95478ffbb566e45f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:01 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:35194
I20260620 15:17:01.714771 7588 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ea36abb3b4243da95478ffbb566e45f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:01 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:52642
I20260620 15:17:01.737845 7849 tablet_replica.cc:330] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: stopping tablet replica
I20260620 15:17:01.738402 7849 raft_consensus.cc:2229] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:01.738996 7849 raft_consensus.cc:2258] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:01.740260 7850 tablet_replica.cc:330] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: stopping tablet replica
I20260620 15:17:01.740708 7850 raft_consensus.cc:2229] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:01.741238 7850 raft_consensus.cc:2258] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:01.741667 7849 ts_tablet_manager.cc:1905] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:01.742509 7851 tablet_replica.cc:330] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: stopping tablet replica
I20260620 15:17:01.742813 7851 raft_consensus.cc:2229] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:17:01.743333 7851 raft_consensus.cc:2258] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:01.745800 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 6502
I20260620 15:17:01.750499 7850 ts_tablet_manager.cc:1905] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:01.754155 6564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:01.766894 7849 ts_tablet_manager.cc:1918] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:01.767076 7849 log.cc:1196] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/6ea36abb3b4243da95478ffbb566e45f
I20260620 15:17:01.767637 7849 ts_tablet_manager.cc:1939] T 6ea36abb3b4243da95478ffbb566e45f P fe55b5749b6843c899d0d2d8896c31cf: Deleting consensus metadata
I20260620 15:17:01.771358 7850 ts_tablet_manager.cc:1918] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:01.771523 7850 log.cc:1196] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/6ea36abb3b4243da95478ffbb566e45f
I20260620 15:17:01.771670 7851 ts_tablet_manager.cc:1905] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:01.772131 7850 ts_tablet_manager.cc:1939] T 6ea36abb3b4243da95478ffbb566e45f P 22559783906942448d1eec02ed6331fe: Deleting consensus metadata
I20260620 15:17:01.783068 7851 ts_tablet_manager.cc:1918] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:01.783335 7851 log.cc:1196] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/6ea36abb3b4243da95478ffbb566e45f
I20260620 15:17:01.783985 7851 ts_tablet_manager.cc:1939] T 6ea36abb3b4243da95478ffbb566e45f P d3c3f0c9d7ff4262a9fab6dd8d53204b: Deleting consensus metadata
I20260620 15:17:01.806425 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.716528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.0.97.125:37799)
0620 15:17:01.737202 (+ 20674us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.737212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.742040 (+ 4828us) server_negotiation.cc:407] Connection header received
0620 15:17:01.743993 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.744001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.744100 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.744184 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:01.758889 (+ 14705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.761868 (+ 2979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.765463 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.766359 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.766489 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.778174 (+ 11685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:01.778183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:01.778192 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:01.778245 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:01.781961 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.783602 (+ 1641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.783608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.783611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.783768 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.796989 (+ 13221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.796996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.797000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.797401 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:01.805594 (+ 8193us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.805745 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.806010 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20483,"thread_start_us":20700,"threads_started":1}
I20260620 15:17:01.817247 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.744698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42568 (local address 127.0.97.124:42865)
0620 15:17:01.765233 (+ 20535us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.765242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.798884 (+ 33642us) server_negotiation.cc:407] Connection header received
0620 15:17:01.798955 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.798960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.799058 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.799168 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:01.801172 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.802458 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.804476 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.805233 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.805340 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.810368 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:01.810374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:01.810379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:01.810415 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:01.813826 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.814613 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.814618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.814622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.814751 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.815156 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.815161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.815164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.815472 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:01.815681 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.816644 (+ 963us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.816860 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20363,"thread_start_us":20194,"threads_started":1}
I20260620 15:17:01.915630 6518 catalog_manager.cc:4841] TS d3c3f0c9d7ff4262a9fab6dd8d53204b (127.0.97.66:37201): tablet 6ea36abb3b4243da95478ffbb566e45f (table test-table [id=7b4ac1716948432b83d161d3e04a1517]) successfully deleted
I20260620 15:17:01.916926 6521 catalog_manager.cc:4841] TS fe55b5749b6843c899d0d2d8896c31cf (127.0.97.67:39053): tablet 6ea36abb3b4243da95478ffbb566e45f (table test-table [id=7b4ac1716948432b83d161d3e04a1517]) successfully deleted
I20260620 15:17:01.916965 6518 catalog_manager.cc:4841] TS 22559783906942448d1eec02ed6331fe (127.0.97.65:33143): tablet 6ea36abb3b4243da95478ffbb566e45f (table test-table [id=7b4ac1716948432b83d161d3e04a1517]) successfully deleted
I20260620 15:17:01.917533 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 6502
I20260620 15:17:01.963433 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.942702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.0.97.125:37799)
0620 15:17:01.942916 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.942925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.943923 (+ 998us) server_negotiation.cc:407] Connection header received
0620 15:17:01.944463 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.944472 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.944669 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.944765 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:01.947959 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.949443 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.952081 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.952935 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.953068 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.957055 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:01.957063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:01.957070 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:01.957116 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:01.960746 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.961496 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.961503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.961508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.961653 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.962061 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.962066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.962070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.962328 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:01.962590 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.962892 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.963093 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:17:01.972021 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:01.943911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.0.97.124:42865)
0620 15:17:01.944127 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:01.944136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:01.955089 (+ 10953us) server_negotiation.cc:407] Connection header received
0620 15:17:01.955246 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:01.955251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:01.955331 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:01.955418 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:01.958997 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.960185 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.962063 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:01.962720 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:01.962845 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:01.966143 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:01.966150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:01.966155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:01.966190 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:01.969441 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.970157 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.970162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.970166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.970294 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:01.970645 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:01.970650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:01.970653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:01.970960 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:01.971208 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:01.971493 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 15:17:01.971692 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:17:02.697892 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:02.696809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.67:43017)
0620 15:17:02.697474 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:02.697622 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":326,"threads_started":1}
W20260620 15:17:02.698372 7795 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:46541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
I20260620 15:17:02.700271 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:02.699464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.66:59111)
0620 15:17:02.699973 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:02.700073 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":484,"threads_started":1}
W20260620 15:17:02.700685 7654 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:46541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
I20260620 15:17:02.705864 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:02.705222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.65:33127)
0620 15:17:02.705641 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:02.705735 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":227,"threads_started":1}
W20260620 15:17:02.706238 7507 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:46541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
I20260620 15:17:03.212929 7866 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bca18fbf31a04a8ca99055a3367b1992)
I20260620 15:17:03.213033 7867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bca18fbf31a04a8ca99055a3367b1992)
I20260620 15:17:03.213128 7866 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } }
I20260620 15:17:03.213284 7867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } }
I20260620 15:17:03.213989 7866 leader_election.cc:288] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bca18fbf31a04a8ca99055a3367b1992 (127.0.97.126:46541), 0c419a33235d428ba328b43e6fe45a1e (127.0.97.125:37799)
I20260620 15:17:03.214211 7867 leader_election.cc:288] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bca18fbf31a04a8ca99055a3367b1992 (127.0.97.126:46541), eb7bb01e759840caa7c9db33aaefe281 (127.0.97.124:42865)
I20260620 15:17:03.227156 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.214256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:44304)
0620 15:17:03.226691 (+ 12435us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.226904 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12314,"thread_start_us":11861,"threads_started":1}
I20260620 15:17:03.227174 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.214674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:44312)
0620 15:17:03.226688 (+ 12014us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.226872 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11904,"threads_started":1}
W20260620 15:17:03.227469 7311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
W20260620 15:17:03.227874 7116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
I20260620 15:17:03.230683 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.229638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:44320)
0620 15:17:03.230207 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.230401 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":299,"threads_started":1}
W20260620 15:17:03.230931 7116 leader_election.cc:334] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bca18fbf31a04a8ca99055a3367b1992 (127.0.97.126:46541): Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
I20260620 15:17:03.231017 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.230567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:44334)
0620 15:17:03.230760 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.230839 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 15:17:03.231190 7311 leader_election.cc:334] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bca18fbf31a04a8ca99055a3367b1992 (127.0.97.126:46541): Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
I20260620 15:17:03.234853 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.214246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:37799 (local address 127.0.0.1:58252)
0620 15:17:03.226451 (+ 12205us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.226476 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:03.226655 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:03.228340 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:03.228345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:03.228373 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:03.228821 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:03.228838 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.230805 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.230810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:03.232524 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:03.232543 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.234314 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.234319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:03.234438 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.234461 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:17:03.234529 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 15:17:03.234605 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12038,"mutex_wait_us":2,"thread_start_us":12516,"threads_started":1}
I20260620 15:17:03.235069 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.214499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58252 (local address 127.0.97.125:37799)
0620 15:17:03.227693 (+ 13194us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:03.227698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:03.227736 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:17:03.227803 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:03.227808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:03.227923 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:03.228070 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:03.229103 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.230657 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.232696 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.234166 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.234295 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.234385 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:03.234588 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 15:17:03.234724 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":13077,"thread_start_us":12887,"threads_started":1}
I20260620 15:17:03.235620 7142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb7bb01e759840caa7c9db33aaefe281" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0c419a33235d428ba328b43e6fe45a1e" is_pre_election: true
I20260620 15:17:03.235867 7142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb7bb01e759840caa7c9db33aaefe281 in term 1.
I20260620 15:17:03.236348 7313 leader_election.cc:302] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c419a33235d428ba328b43e6fe45a1e, eb7bb01e759840caa7c9db33aaefe281; no voters: bca18fbf31a04a8ca99055a3367b1992
I20260620 15:17:03.236716 7866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 15:17:03.236833 7866 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bca18fbf31a04a8ca99055a3367b1992)
I20260620 15:17:03.236934 7866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:17:03.237049 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.227578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:42865 (local address 127.0.0.1:42582)
0620 15:17:03.227800 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.227817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:03.227931 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:03.230850 (+ 2919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:03.230854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:03.230895 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:03.231217 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:03.231231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.232589 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.232597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:03.234804 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:03.234822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.236287 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.236293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:03.236525 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.236567 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 15:17:03.236652 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:17:03.236771 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":91}
I20260620 15:17:03.237198 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.228461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42582 (local address 127.0.97.124:42865)
0620 15:17:03.230558 (+ 2097us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:03.230563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:03.230585 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:17:03.230638 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:03.230643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:03.230710 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:03.231360 (+ 650us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:03.231610 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.232440 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.234963 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.236132 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.236249 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.236342 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:03.236690 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 15:17:03.236827 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1977,"thread_start_us":1845,"threads_started":1}
I20260620 15:17:03.237794 7337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c419a33235d428ba328b43e6fe45a1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb7bb01e759840caa7c9db33aaefe281" is_pre_election: true
I20260620 15:17:03.239512 7866 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } }
I20260620 15:17:03.239956 7337 raft_consensus.cc:2379] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c419a33235d428ba328b43e6fe45a1e in current term 2: Already voted for candidate eb7bb01e759840caa7c9db33aaefe281 in this term.
I20260620 15:17:03.240146 7866 leader_election.cc:288] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [CANDIDATE]: Term 2 election: Requested vote from peers bca18fbf31a04a8ca99055a3367b1992 (127.0.97.126:46541), 0c419a33235d428ba328b43e6fe45a1e (127.0.97.125:37799)
I20260620 15:17:03.240612 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.240172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:44346)
0620 15:17:03.240353 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.240425 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 15:17:03.240619 7142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb7bb01e759840caa7c9db33aaefe281" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0c419a33235d428ba328b43e6fe45a1e"
I20260620 15:17:03.240857 7142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:17:03.241031 7116 leader_election.cc:302] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c419a33235d428ba328b43e6fe45a1e; no voters: bca18fbf31a04a8ca99055a3367b1992, eb7bb01e759840caa7c9db33aaefe281
I20260620 15:17:03.241607 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.241246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:44356)
0620 15:17:03.241416 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.241485 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 15:17:03.241758 7311 leader_election.cc:334] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bca18fbf31a04a8ca99055a3367b1992 (127.0.97.126:46541): Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
I20260620 15:17:03.243003 7142 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb7bb01e759840caa7c9db33aaefe281 in term 2.
I20260620 15:17:03.243204 7867 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:17:03.243381 7313 leader_election.cc:302] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c419a33235d428ba328b43e6fe45a1e, eb7bb01e759840caa7c9db33aaefe281; no voters: bca18fbf31a04a8ca99055a3367b1992
I20260620 15:17:03.243665 7866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:17:03.244087 7866 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [term 2 LEADER]: Becoming Leader. State: Replica: eb7bb01e759840caa7c9db33aaefe281, State: Running, Role: LEADER
I20260620 15:17:03.244477 7866 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } }
I20260620 15:17:03.246037 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7bb01e759840caa7c9db33aaefe281. Latest consensus state: current_term: 2 leader_uuid: "eb7bb01e759840caa7c9db33aaefe281" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:03.246335 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:03.247004 7878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:17:03.250735 7878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:17:03.251602 7878 catalog_manager.cc:1259] Loaded cluster ID: b14ee8ff09174c258de5f70796d4f926
I20260620 15:17:03.251669 7878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:17:03.252836 7878 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:17:03.253636 7878 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Loaded TSK: 0
I20260620 15:17:03.254432 7878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:17:03.306737 7326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:17:03.313937 7326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:17:03.318267 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.317887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:44396)
0620 15:17:03.318068 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.318151 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 15:17:03.318253 7142 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [term 2 FOLLOWER]: Refusing update from remote peer eb7bb01e759840caa7c9db33aaefe281: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 15:17:03.318931 7876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 15:17:03.319331 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.318959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:44404)
0620 15:17:03.319110 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.319187 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 15:17:03.319521 7311 consensus_peers.cc:487] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 -> Peer bca18fbf31a04a8ca99055a3367b1992 (127.0.97.126:46541): Couldn't send request to peer bca18fbf31a04a8ca99055a3367b1992. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 15:17:03.321946 7867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb7bb01e759840caa7c9db33aaefe281. Latest consensus state: current_term: 2 leader_uuid: "eb7bb01e759840caa7c9db33aaefe281" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:03.322217 7867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:17:03.323571 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "eb7bb01e759840caa7c9db33aaefe281" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:03.323897 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281 [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:03.324107 7867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "eb7bb01e759840caa7c9db33aaefe281" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bca18fbf31a04a8ca99055a3367b1992" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46541 } } peers { permanent_uuid: "0c419a33235d428ba328b43e6fe45a1e" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 37799 } attrs { promote: false } } peers { permanent_uuid: "eb7bb01e759840caa7c9db33aaefe281" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 42865 } attrs { promote: false } } }
I20260620 15:17:03.324388 7867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c419a33235d428ba328b43e6fe45a1e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:17:03.339620 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.332301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:33143 (local address 127.0.0.1:38336)
0620 15:17:03.332434 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.332451 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:03.332609 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:03.333276 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:03.333281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:03.333302 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:03.333569 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:03.333577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.335487 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.335492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:03.336832 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:03.336847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.338884 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.338889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:03.339060 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.339095 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:17:03.339197 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:17:03.339332 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 15:17:03.339854 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.332394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.0.97.65:33143)
0620 15:17:03.333020 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:03.333025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:03.333038 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:17:03.333090 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:03.333093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:03.333145 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:03.333255 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:03.333711 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.335341 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.337030 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.338639 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.338807 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.338966 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:03.339262 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 15:17:03.339495 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":440,"threads_started":1}
I20260620 15:17:03.340546 7441 tablet_service.cc:1467] Processing CreateTablet for tablet c9272f3243b44178aba957ec16061fd4 (DEFAULT_TABLE table=test-table [id=406b902923984721a2fd2e19cdaafa2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:03.340992 7441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9272f3243b44178aba957ec16061fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:03.343225 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.331363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39053 (local address 127.0.0.1:35200)
0620 15:17:03.331491 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.331510 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:03.331608 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:03.337153 (+ 5545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:03.337158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:03.337178 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:03.337317 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:03.337327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.338873 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.338878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:03.340704 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:03.340718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.342507 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.342513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:03.342697 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.342728 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:17:03.342829 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:17:03.342960 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 15:17:03.343253 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.331471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.0.97.67:39053)
0620 15:17:03.336842 (+ 5371us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:03.336847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:03.336867 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:17:03.336936 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:03.336941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:03.337038 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:03.337141 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:03.337587 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.338697 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.340897 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.342334 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.342458 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.342558 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:03.342866 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 15:17:03.342993 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5256,"thread_start_us":5172,"threads_started":1}
I20260620 15:17:03.344013 7729 tablet_service.cc:1467] Processing CreateTablet for tablet c9272f3243b44178aba957ec16061fd4 (DEFAULT_TABLE table=test-table [id=406b902923984721a2fd2e19cdaafa2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:03.344434 7729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9272f3243b44178aba957ec16061fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:03.347404 7889 tablet_bootstrap.cc:492] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe: Bootstrap starting.
I20260620 15:17:03.348179 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.332769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.0.97.66:37201)
0620 15:17:03.340889 (+ 8120us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:03.340893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:03.340907 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:17:03.340961 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:03.340965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:03.341014 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:03.341127 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:03.341819 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.342840 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.345387 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.346628 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.346792 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.346946 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:03.347713 (+ 767us) server_negotiation.cc:299] Negotiation successful
0620 15:17:03.347871 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8024,"thread_start_us":7867,"threads_started":1}
I20260620 15:17:03.348905 7889 tablet_bootstrap.cc:654] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:03.348940 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.333068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37201 (local address 127.0.0.1:52656)
0620 15:17:03.334084 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.334100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:03.334193 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:03.341306 (+ 7113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:03.341310 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:03.341332 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:03.341675 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:03.341687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.342989 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.342993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:03.345196 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:03.345208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.347288 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.347295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:03.347504 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.347540 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:17:03.348442 (+ 902us) client_negotiation.cc:240] Negotiation successful
0620 15:17:03.348601 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"thread_start_us":654,"threads_started":1}
I20260620 15:17:03.349834 7588 tablet_service.cc:1467] Processing CreateTablet for tablet c9272f3243b44178aba957ec16061fd4 (DEFAULT_TABLE table=test-table [id=406b902923984721a2fd2e19cdaafa2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:03.350215 7588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9272f3243b44178aba957ec16061fd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:03.350878 7891 tablet_bootstrap.cc:492] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf: Bootstrap starting.
I20260620 15:17:03.351157 7889 tablet_bootstrap.cc:492] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe: No bootstrap required, opened a new log
I20260620 15:17:03.351301 7889 ts_tablet_manager.cc:1397] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:17:03.352047 7889 raft_consensus.cc:348] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } }
I20260620 15:17:03.352250 7889 raft_consensus.cc:374] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:03.352332 7889 raft_consensus.cc:729] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22559783906942448d1eec02ed6331fe, State: Initialized, Role: FOLLOWER
I20260620 15:17:03.352488 7891 tablet_bootstrap.cc:654] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:03.352602 7889 consensus_queue.cc:260] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } }
I20260620 15:17:03.354523 7891 tablet_bootstrap.cc:492] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf: No bootstrap required, opened a new log
I20260620 15:17:03.354681 7891 ts_tablet_manager.cc:1397] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:17:03.355307 7891 raft_consensus.cc:348] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } }
I20260620 15:17:03.355516 7891 raft_consensus.cc:374] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:03.355618 7891 raft_consensus.cc:729] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe55b5749b6843c899d0d2d8896c31cf, State: Initialized, Role: FOLLOWER
I20260620 15:17:03.355894 7891 consensus_queue.cc:260] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } }
I20260620 15:17:03.358189 7895 tablet_bootstrap.cc:492] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b: Bootstrap starting.
I20260620 15:17:03.358791 7891 ts_tablet_manager.cc:1428] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260620 15:17:03.360282 7895 tablet_bootstrap.cc:654] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:03.362383 7895 tablet_bootstrap.cc:492] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b: No bootstrap required, opened a new log
I20260620 15:17:03.362665 7895 ts_tablet_manager.cc:1397] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 15:17:03.363421 7895 raft_consensus.cc:348] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } }
I20260620 15:17:03.363632 7895 raft_consensus.cc:374] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:03.363747 7895 raft_consensus.cc:729] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c3f0c9d7ff4262a9fab6dd8d53204b, State: Initialized, Role: FOLLOWER
I20260620 15:17:03.363974 7895 consensus_queue.cc:260] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } }
I20260620 15:17:03.364763 7895 ts_tablet_manager.cc:1428] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 15:17:03.371315 7889 ts_tablet_manager.cc:1428] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe: Time spent starting tablet: real 0.020s user 0.004s sys 0.004s
I20260620 15:17:03.376905 7506 heartbeater.cc:503] Master 127.0.97.124:42865 was elected leader, sending a full tablet report...
I20260620 15:17:03.377295 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.359870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.65:38349)
0620 15:17:03.376735 (+ 16865us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.376855 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16728,"thread_start_us":16579,"threads_started":1}
I20260620 15:17:03.378043 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.365360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.66:52727)
0620 15:17:03.377691 (+ 12331us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.377768 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12241,"threads_started":1}
I20260620 15:17:03.378539 7653 heartbeater.cc:503] Master 127.0.97.124:42865 was elected leader, sending a full tablet report...
I20260620 15:17:03.379105 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.378054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.65:50551)
0620 15:17:03.378817 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.378920 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641,"threads_started":1}
I20260620 15:17:03.379848 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.379126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.66:42921)
0620 15:17:03.379637 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.379716 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400}
I20260620 15:17:03.380995 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.359291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.67:40587)
0620 15:17:03.380453 (+ 21162us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.380645 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20588,"thread_start_us":20794,"threads_started":1}
I20260620 15:17:03.382437 7794 heartbeater.cc:503] Master 127.0.97.124:42865 was elected leader, sending a full tablet report...
I20260620 15:17:03.382652 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.381649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.67:35377)
0620 15:17:03.382139 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.382337 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":264,"threads_started":1}
I20260620 15:17:03.518803 7899 raft_consensus.cc:482] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:17:03.519021 7899 raft_consensus.cc:504] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } }
I20260620 15:17:03.519769 7899 leader_election.cc:288] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe55b5749b6843c899d0d2d8896c31cf (127.0.97.67:39053), 22559783906942448d1eec02ed6331fe (127.0.97.65:33143)
I20260620 15:17:03.520300 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9272f3243b44178aba957ec16061fd4" candidate_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22559783906942448d1eec02ed6331fe" is_pre_election: true
I20260620 15:17:03.520277 7748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9272f3243b44178aba957ec16061fd4" candidate_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe55b5749b6843c899d0d2d8896c31cf" is_pre_election: true
I20260620 15:17:03.520576 7461 raft_consensus.cc:2454] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3c3f0c9d7ff4262a9fab6dd8d53204b in term 0.
I20260620 15:17:03.520596 7748 raft_consensus.cc:2454] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3c3f0c9d7ff4262a9fab6dd8d53204b in term 0.
I20260620 15:17:03.521041 7541 leader_election.cc:302] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22559783906942448d1eec02ed6331fe, d3c3f0c9d7ff4262a9fab6dd8d53204b; no voters:
I20260620 15:17:03.521303 7899 raft_consensus.cc:2781] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:17:03.521416 7899 raft_consensus.cc:482] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:17:03.521499 7899 raft_consensus.cc:3037] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:03.523447 7899 raft_consensus.cc:504] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } }
I20260620 15:17:03.523900 7899 leader_election.cc:288] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [CANDIDATE]: Term 1 election: Requested vote from peers fe55b5749b6843c899d0d2d8896c31cf (127.0.97.67:39053), 22559783906942448d1eec02ed6331fe (127.0.97.65:33143)
I20260620 15:17:03.524318 7748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9272f3243b44178aba957ec16061fd4" candidate_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe55b5749b6843c899d0d2d8896c31cf"
I20260620 15:17:03.524484 7748 raft_consensus.cc:3037] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:03.524385 7461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9272f3243b44178aba957ec16061fd4" candidate_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22559783906942448d1eec02ed6331fe"
I20260620 15:17:03.524663 7461 raft_consensus.cc:3037] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:03.526472 7748 raft_consensus.cc:2454] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3c3f0c9d7ff4262a9fab6dd8d53204b in term 1.
I20260620 15:17:03.526636 7461 raft_consensus.cc:2454] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3c3f0c9d7ff4262a9fab6dd8d53204b in term 1.
I20260620 15:17:03.526872 7544 leader_election.cc:302] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3c3f0c9d7ff4262a9fab6dd8d53204b, fe55b5749b6843c899d0d2d8896c31cf; no voters:
I20260620 15:17:03.527177 7899 raft_consensus.cc:2781] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:03.527580 7899 raft_consensus.cc:686] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 1 LEADER]: Becoming Leader. State: Replica: d3c3f0c9d7ff4262a9fab6dd8d53204b, State: Running, Role: LEADER
I20260620 15:17:03.527902 7899 consensus_queue.cc:237] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } }
I20260620 15:17:03.528637 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.528160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.66:40901)
0620 15:17:03.528302 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.528391 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 15:17:03.530519 7326 catalog_manager.cc:5495] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b reported cstate change: term changed from 0 to 1, leader changed from <none> to d3c3f0c9d7ff4262a9fab6dd8d53204b (127.0.97.66). New cstate: current_term: 1 leader_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3c3f0c9d7ff4262a9fab6dd8d53204b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37201 } health_report { overall_health: HEALTHY } } }
I20260620 15:17:03.576942 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.571596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.0.97.66:37201)
0620 15:17:03.571772 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:03.571776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:03.571792 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:17:03.571922 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:03.571925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:03.571984 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:03.572062 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:17:03.572374 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.573251 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.574895 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.575514 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.575649 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.576121 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:17:03.576216 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:17:03.576307 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:03.576461 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 15:17:03.576562 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:17:03.579890 7461 raft_consensus.cc:1261] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe [term 1 FOLLOWER]: Refusing update from remote peer d3c3f0c9d7ff4262a9fab6dd8d53204b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:03.579924 7748 raft_consensus.cc:1261] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf [term 1 FOLLOWER]: Refusing update from remote peer d3c3f0c9d7ff4262a9fab6dd8d53204b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:03.580534 7903 consensus_queue.cc:1035] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe55b5749b6843c899d0d2d8896c31cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:03.580852 7899 consensus_queue.cc:1035] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [LEADER]: Connected to new peer: Peer: permanent_uuid: "22559783906942448d1eec02ed6331fe" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 33143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:03.582746 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.582341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.67:42413)
0620 15:17:03.582502 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.582610 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:17:03.584272 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.583534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.65:36461)
0620 15:17:03.583845 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.584019 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 15:17:03.588403 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.587918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.66:46231)
0620 15:17:03.588175 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.588261 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 15:17:03.589789 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.589178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.67:58355)
0620 15:17:03.589535 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.589632 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
I20260620 15:17:03.595691 7326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:03.595999 7326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:03.600360 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.599891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.66:50933)
0620 15:17:03.600123 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.600209 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 15:17:03.601791 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.600802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.97.65:58359)
0620 15:17:03.601514 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.601628 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599}
I20260620 15:17:03.605203 7326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb7bb01e759840caa7c9db33aaefe281: Sending DeleteTablet for 3 replicas of tablet c9272f3243b44178aba957ec16061fd4
I20260620 15:17:03.606236 7441 tablet_service.cc:1514] Processing DeleteTablet for tablet c9272f3243b44178aba957ec16061fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:03 UTC) from {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:38336
I20260620 15:17:03.606451 7440 tablet_service.cc:1514] Processing DeleteTablet for tablet c9272f3243b44178aba957ec16061fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:03 UTC) from {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:38336
I20260620 15:17:03.606720 7588 tablet_service.cc:1514] Processing DeleteTablet for tablet c9272f3243b44178aba957ec16061fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:03 UTC) from {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:52656
I20260620 15:17:03.607096 7587 tablet_service.cc:1514] Processing DeleteTablet for tablet c9272f3243b44178aba957ec16061fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:03 UTC) from {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:52656
I20260620 15:17:03.607443 7915 tablet_replica.cc:330] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe: stopping tablet replica
I20260620 15:17:03.607581 7916 tablet_replica.cc:330] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b: stopping tablet replica
I20260620 15:17:03.607682 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 7306
I20260620 15:17:03.607728 7916 raft_consensus.cc:2229] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:17:03.608065 7915 raft_consensus.cc:2229] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:03.608094 7916 raft_consensus.cc:2258] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:03.608316 7915 raft_consensus.cc:2258] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:03.609489 7915 ts_tablet_manager.cc:1905] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:03.609489 7916 ts_tablet_manager.cc:1905] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:03.610273 7729 tablet_service.cc:1514] Processing DeleteTablet for tablet c9272f3243b44178aba957ec16061fd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:03 UTC) from {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:35200
I20260620 15:17:03.615932 7916 ts_tablet_manager.cc:1918] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:03.616070 7916 log.cc:1196] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/c9272f3243b44178aba957ec16061fd4
I20260620 15:17:03.616185 7915 ts_tablet_manager.cc:1918] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:03.616308 7915 log.cc:1196] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/c9272f3243b44178aba957ec16061fd4
I20260620 15:17:03.616549 7916 ts_tablet_manager.cc:1939] T c9272f3243b44178aba957ec16061fd4 P d3c3f0c9d7ff4262a9fab6dd8d53204b: Deleting consensus metadata
I20260620 15:17:03.616747 7915 ts_tablet_manager.cc:1939] T c9272f3243b44178aba957ec16061fd4 P 22559783906942448d1eec02ed6331fe: Deleting consensus metadata
I20260620 15:17:03.617544 7311 catalog_manager.cc:4841] TS d3c3f0c9d7ff4262a9fab6dd8d53204b (127.0.97.66:37201): tablet c9272f3243b44178aba957ec16061fd4 (table test-table [id=406b902923984721a2fd2e19cdaafa2c]) successfully deleted
I20260620 15:17:03.618116 7311 catalog_manager.cc:4841] TS 22559783906942448d1eec02ed6331fe (127.0.97.65:33143): tablet c9272f3243b44178aba957ec16061fd4 (table test-table [id=406b902923984721a2fd2e19cdaafa2c]) successfully deleted
W20260620 15:17:03.618567 7311 catalog_manager.cc:4808] TS 22559783906942448d1eec02ed6331fe (127.0.97.65:33143): delete failed for tablet c9272f3243b44178aba957ec16061fd4 because the tablet was not found. No further retry: Not found: Tablet not found: c9272f3243b44178aba957ec16061fd4
W20260620 15:17:03.620631 7311 catalog_manager.cc:4808] TS d3c3f0c9d7ff4262a9fab6dd8d53204b (127.0.97.66:37201): delete failed for tablet c9272f3243b44178aba957ec16061fd4 because the tablet was not found. No further retry: Not found: Tablet not found: c9272f3243b44178aba957ec16061fd4
I20260620 15:17:03.621098 7918 tablet_replica.cc:330] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf: stopping tablet replica
I20260620 15:17:03.621281 7918 raft_consensus.cc:2229] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:03.621541 7918 raft_consensus.cc:2258] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:03.622354 7918 ts_tablet_manager.cc:1905] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:03.626973 7918 ts_tablet_manager.cc:1918] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:03.627147 7918 log.cc:1196] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/c9272f3243b44178aba957ec16061fd4
I20260620 15:17:03.627506 7918 ts_tablet_manager.cc:1939] T c9272f3243b44178aba957ec16061fd4 P fe55b5749b6843c899d0d2d8896c31cf: Deleting consensus metadata
I20260620 15:17:03.628304 7314 catalog_manager.cc:4841] TS fe55b5749b6843c899d0d2d8896c31cf (127.0.97.67:39053): tablet c9272f3243b44178aba957ec16061fd4 (table test-table [id=406b902923984721a2fd2e19cdaafa2c]) successfully deleted
I20260620 15:17:03.630789 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.609135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.0.97.65:33143)
0620 15:17:03.609303 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:03.609309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:03.618101 (+ 8792us) server_negotiation.cc:407] Connection header received
0620 15:17:03.618332 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:03.618338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:03.618426 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:03.618496 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:03.620013 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.621090 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.622452 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.622867 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.622992 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.625431 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:03.625450 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:03.625464 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:03.625498 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:03.628244 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:03.628936 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:03.628944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:03.628946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:03.629060 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:03.629444 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:03.629454 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:03.629456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:03.629754 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:03.630063 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:03.630308 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 15:17:03.630512 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:17:03.631336 7501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:03.722254 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 7306
I20260620 15:17:03.739414 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 7518
I20260620 15:17:03.752189 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.740154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.0.97.66:37201)
0620 15:17:03.740335 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:03.740342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:03.740446 (+ 104us) server_negotiation.cc:407] Connection header received
0620 15:17:03.740675 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:03.740680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:03.740759 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:03.740871 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:03.742084 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.742803 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.744065 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.744460 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.744621 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.746993 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:03.747018 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:03.747033 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:03.747062 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:03.749633 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:03.750275 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:03.750286 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:03.750294 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:03.750414 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:03.750766 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:03.750773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:03.750776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:03.751107 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:03.751394 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:03.751689 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 15:17:03.751903 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:17:03.752933 7648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:03.863665 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.862926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:44414)
0620 15:17:03.863382 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.863471 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":245,"threads_started":1}
I20260620 15:17:03.865056 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.864710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46541 (local address 127.0.0.1:44430)
0620 15:17:03.864859 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:03.864947 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 15:17:03.870733 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 7518
I20260620 15:17:03.893076 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 7666
I20260620 15:17:03.906383 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:03.893755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.0.97.67:39053)
0620 15:17:03.894217 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:03.894221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:03.894237 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:17:03.894287 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:03.894290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:03.894347 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:03.894495 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:03.895863 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.896751 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.898013 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:03.898404 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:03.898498 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:03.901087 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:03.901100 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:03.901116 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:03.901166 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:03.904063 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:03.904706 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:03.904713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:03.904715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:03.904838 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:03.905160 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:03.905164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:03.905167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:03.905483 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:03.905693 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:03.905881 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 15:17:03.906035 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":263,"threads_started":1}
I20260620 15:17:03.907084 7789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:03.998584 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 7666
I20260620 15:17:04.016598 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 6995
I20260620 15:17:04.017879 7162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:04.131481 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 6995
I20260620 15:17:04.151659 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 7096
I20260620 15:17:04.152694 7357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:04.269724 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 7096
2026-06-20T15:17:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9455 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:17:04 dist-test-slave-7s64 krb5kdc[7929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:17:04 dist-test-slave-7s64 krb5kdc[7929](info): set up 2 sockets
Jun 20 15:17:04 dist-test-slave-7s64 krb5kdc[7929](info): commencing operation
krb5kdc: starting...
W20260620 15:17:05.361449 389 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:17:05 dist-test-slave-7s64 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:17:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:17:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.126.keytab.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.126.keytab.
I20260620 15:17:05.551980 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38049
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:05.860653 7942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:05.860970 7942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:05.861124 7942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:05.868009 7942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:17:05.868134 7942 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:17:05.868222 7942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:05.868343 7942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:05.868424 7942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:17:05.868520 7942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:17:05.877760 7942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38049
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:05.878966 7942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:05.880239 7942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:05.891489 7947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:05.891595 7950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:05.891665 7948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:05.892796 7942 server_base.cc:1034] running on GCE node
I20260620 15:17:05.893486 7942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:05.894740 7942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:05.895987 7942 hybrid_clock.cc:648] HybridClock initialized: now 1781968625895935 us; error 49 us; skew 500 ppm
I20260620 15:17:05.896318 7942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:05 dist-test-slave-7s64 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968625, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:05.899356 7942 init.cc:377] Logged in from keytab as oryx/127.0.97.126@KRBTEST.COM (short username oryx)
I20260620 15:17:05.900682 7942 webserver.cc:460] Webserver started at http://127.0.97.126:42095/ using document root <none> and password file <none>
I20260620 15:17:05.901295 7942 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:05.901376 7942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:05.901623 7942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:05.904274 7942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "f15bd72a16204aa4818d02a2e96245da"
format_stamp: "Formatted at 2026-06-20 15:17:05 on dist-test-slave-7s64"
server_key: "1b8fb0d163c1c6a49436400bf02fcf59"
server_key_iv: "bf12b8db92ec11ae3b4ddb67470bca94"
server_key_version: "encryptionkey@0"
I20260620 15:17:05.905030 7942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "f15bd72a16204aa4818d02a2e96245da"
format_stamp: "Formatted at 2026-06-20 15:17:05 on dist-test-slave-7s64"
server_key: "1b8fb0d163c1c6a49436400bf02fcf59"
server_key_iv: "bf12b8db92ec11ae3b4ddb67470bca94"
server_key_version: "encryptionkey@0"
I20260620 15:17:05.909489 7942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:17:05.912616 7957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:05.913793 7942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:17:05.913913 7942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "f15bd72a16204aa4818d02a2e96245da"
format_stamp: "Formatted at 2026-06-20 15:17:05 on dist-test-slave-7s64"
server_key: "1b8fb0d163c1c6a49436400bf02fcf59"
server_key_iv: "bf12b8db92ec11ae3b4ddb67470bca94"
server_key_version: "encryptionkey@0"
I20260620 15:17:05.914093 7942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:05.932729 7942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:05.936640 7942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:05.936825 7942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:05.957450 7942 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:38049
I20260620 15:17:05.957469 8018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:38049 every 8 connection(s)
I20260620 15:17:05.958693 7942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:17:05.962188 8019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:05.968112 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 7942
I20260620 15:17:05.968322 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:17:05.968963 389 external_mini_cluster.cc:1428] Setting key 31a59afb49ebec8ebe1c6a21da05e573
I20260620 15:17:05.971992 8019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da: Bootstrap starting.
I20260620 15:17:05.974967 8019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:05.975945 8019 log.cc:824] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:05.978510 8019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da: No bootstrap required, opened a new log
Jun 20 15:17:05 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:17:05.986454 8019 raft_consensus.cc:348] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15bd72a16204aa4818d02a2e96245da" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38049 } }
I20260620 15:17:05.986799 8019 raft_consensus.cc:374] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:05.986933 8019 raft_consensus.cc:729] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15bd72a16204aa4818d02a2e96245da, State: Initialized, Role: FOLLOWER
I20260620 15:17:05.987534 8019 consensus_queue.cc:260] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15bd72a16204aa4818d02a2e96245da" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38049 } }
I20260620 15:17:05.987808 8019 raft_consensus.cc:388] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:17:05.987941 8019 raft_consensus.cc:482] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:17:05.988125 8019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:05.990613 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:05.970773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.0.97.126:38049)
0620 15:17:05.971487 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:05.971497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:05.971534 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:17:05.973274 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:05.973298 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:05.973646 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:05.974033 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:05.975420 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:05.976789 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:05.978015 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:05.978742 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:05.978937 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:05.983041 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:05.983048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:05.983065 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:05.983103 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:05.986015 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:05.986668 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:05.986682 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:05.986687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:05.986947 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:05.987375 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:05.987380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:05.987383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:05.988078 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:05.988386 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:05.988825 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 15:17:05.989278 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":344,"threads_started":1}
I20260620 15:17:05.991420 8019 raft_consensus.cc:504] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15bd72a16204aa4818d02a2e96245da" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38049 } }
I20260620 15:17:05.991946 8019 leader_election.cc:302] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f15bd72a16204aa4818d02a2e96245da; no voters:
I20260620 15:17:05.992570 8019 leader_election.cc:288] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:17:05.992666 8024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:05.993757 8024 raft_consensus.cc:686] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [term 1 LEADER]: Becoming Leader. State: Replica: f15bd72a16204aa4818d02a2e96245da, State: Running, Role: LEADER
I20260620 15:17:05.994318 8024 consensus_queue.cc:237] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15bd72a16204aa4818d02a2e96245da" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38049 } }
I20260620 15:17:05.994671 8019 sys_catalog.cc:564] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:17:05.997323 8025 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f15bd72a16204aa4818d02a2e96245da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15bd72a16204aa4818d02a2e96245da" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38049 } } }
I20260620 15:17:05.997411 8026 sys_catalog.cc:455] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [sys.catalog]: SysCatalogTable state changed. Reason: New leader f15bd72a16204aa4818d02a2e96245da. Latest consensus state: current_term: 1 leader_uuid: "f15bd72a16204aa4818d02a2e96245da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f15bd72a16204aa4818d02a2e96245da" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38049 } } }
I20260620 15:17:05.997669 8026 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:05.997668 8025 sys_catalog.cc:458] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:06.000098 8031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:17:06.004390 8031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:17:06.012109 8031 catalog_manager.cc:1347] Generated new cluster ID: e44f3d8d2455462693b3ef1c0fb7d696
I20260620 15:17:06.012235 8031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:17:06.021549 8031 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:17:06.022697 8031 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:17:06.031134 8031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da: Generated new TSK 0
I20260620 15:17:06.031958 8031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab.
I20260620 15:17:06.079583 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38049
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:06.393138 8045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:06.393376 8045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:06.393474 8045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:06.400693 8045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:06.400782 8045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:06.400861 8045 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:17:06.400936 8045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:17:06.409847 8045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:38049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:06.410876 8045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:06.412015 8045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:06.424500 8051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:06.424801 8053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:06.424855 8050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:06.425379 8045 server_base.cc:1034] running on GCE node
I20260620 15:17:06.425937 8045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:06.426805 8045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:06.428064 8045 hybrid_clock.cc:648] HybridClock initialized: now 1781968626428013 us; error 54 us; skew 500 ppm
I20260620 15:17:06.428341 8045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:06 dist-test-slave-7s64 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968626, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:06.431321 8045 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260620 15:17:06.432657 8045 webserver.cc:460] Webserver started at http://127.0.97.65:43629/ using document root <none> and password file <none>
I20260620 15:17:06.433297 8045 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:06.433424 8045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:06.433732 8045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:06.436523 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "a8a97c74af9844d28027d062ec3d7851"
format_stamp: "Formatted at 2026-06-20 15:17:06 on dist-test-slave-7s64"
server_key: "c02bc60b598da41b209ea6d4cc8eff95"
server_key_iv: "c144614436635e719396d9d1e7d75734"
server_key_version: "encryptionkey@0"
I20260620 15:17:06.437294 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "a8a97c74af9844d28027d062ec3d7851"
format_stamp: "Formatted at 2026-06-20 15:17:06 on dist-test-slave-7s64"
server_key: "c02bc60b598da41b209ea6d4cc8eff95"
server_key_iv: "c144614436635e719396d9d1e7d75734"
server_key_version: "encryptionkey@0"
I20260620 15:17:06.442113 8045 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:17:06.445443 8060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:06.446748 8045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:17:06.446954 8045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "a8a97c74af9844d28027d062ec3d7851"
format_stamp: "Formatted at 2026-06-20 15:17:06 on dist-test-slave-7s64"
server_key: "c02bc60b598da41b209ea6d4cc8eff95"
server_key_iv: "c144614436635e719396d9d1e7d75734"
server_key_version: "encryptionkey@0"
I20260620 15:17:06.447175 8045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:06.462473 8045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:06.466372 8045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:06.466622 8045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:06.469063 8045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:06.469169 8045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:06.469326 8045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:06.469403 8045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:17:06 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968626, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:17:06.494854 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:06.469681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:53711 (local address 127.0.97.126:38049)
0620 15:17:06.469946 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:06.469953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:06.471088 (+ 1135us) server_negotiation.cc:407] Connection header received
0620 15:17:06.474004 (+ 2916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:06.474012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:06.474108 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:06.474220 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:06.476740 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.477761 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.480357 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.480995 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.481117 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:06.486817 (+ 5700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:06.486825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:06.486833 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:06.486881 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:06.490564 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:06.491641 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:06.491646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:06.491651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:06.491825 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:06.492518 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:06.492526 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:06.492532 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:06.492909 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:06.493123 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:06.494150 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0620 15:17:06.494416 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:17:06.496629 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:06.470102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38049 (local address 127.0.97.65:53711)
0620 15:17:06.470940 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:06.470984 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:06.473698 (+ 2714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:06.474517 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:06.474527 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:06.475508 (+ 981us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:06.476554 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:06.476582 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.477966 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.477970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:06.480212 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:06.480223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.481425 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.481435 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:06.481747 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:06.482566 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:06.482599 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:06.486540 (+ 3941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:06.490723 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:06.490734 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:06.490758 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:06.491487 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:06.491987 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:06.491994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:06.492000 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:06.492312 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:06.493101 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:06.493112 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:06.493639 (+ 527us) client_negotiation.cc:769] Sending connection context
0620 15:17:06.494026 (+ 387us) client_negotiation.cc:240] Negotiation successful
0620 15:17:06.494720 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":312,"threads_started":1}
I20260620 15:17:06.511929 8045 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:37463
I20260620 15:17:06.511988 8179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:37463 every 8 connection(s)
I20260620 15:17:06.513248 8045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:17:06.515614 8180 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38049
I20260620 15:17:06.515942 8180 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:06.516711 8180 heartbeater.cc:511] Master 127.0.97.126:38049 requested a full tablet report, sending...
I20260620 15:17:06.516966 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 8045
I20260620 15:17:06.517196 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:17:06.518056 389 external_mini_cluster.cc:1428] Setting key ea01ec2173a78e310ab48cfee6a4d5bf
I20260620 15:17:06.519049 7974 ts_manager.cc:194] Registered new tserver with Master: a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463)
I20260620 15:17:06.519464 8160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:06.504586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38049 (local address 127.0.97.65:58301)
0620 15:17:06.505343 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:06.505382 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:06.505540 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:06.505958 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:06.505965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:06.506597 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:06.507113 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:06.507128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.508584 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.508592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:06.510249 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:06.510261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.511301 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.511306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:06.511442 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:06.512189 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:06.512217 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:06.513051 (+ 834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:06.516715 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:06.516721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:06.516727 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:06.517293 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:06.517721 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:06.517728 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:06.517733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:06.517887 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:06.518585 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:06.518589 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:06.518765 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 15:17:06.518917 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 15:17:06.519188 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":437,"threads_started":1}
I20260620 15:17:06.520962 7974 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:53711
I20260620 15:17:06.521039 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:06.504562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:58301 (local address 127.0.97.126:38049)
0620 15:17:06.504761 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:06.504769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:06.505482 (+ 713us) server_negotiation.cc:407] Connection header received
0620 15:17:06.505677 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:06.505685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:06.505798 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:06.505899 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:06.507298 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.508396 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.510418 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.510846 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.511015 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:06.513223 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:06.513231 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:06.513238 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:06.513275 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:06.516553 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:06.517430 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:06.517435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:06.517438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:06.517580 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:06.518035 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:06.518043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:06.518047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:06.518387 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:06.518525 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:06.520426 (+ 1901us) server_negotiation.cc:299] Negotiation successful
0620 15:17:06.520691 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for oryx/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab.
I20260620 15:17:06.561738 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38049
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:06.871042 8184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:06.871275 8184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:06.871430 8184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:06.878334 8184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:06.878446 8184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:06.878599 8184 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:17:06.878715 8184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:17:06.888016 8184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:38049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:06.889189 8184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:06.890452 8184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:06.903131 8192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:06.903110 8190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:06.903621 8189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:06.903713 8184 server_base.cc:1034] running on GCE node
I20260620 15:17:06.904397 8184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:06.905326 8184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:06.906579 8184 hybrid_clock.cc:648] HybridClock initialized: now 1781968626906526 us; error 32 us; skew 500 ppm
I20260620 15:17:06.906883 8184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:06 dist-test-slave-7s64 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968626, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:06.913525 8184 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260620 15:17:06.914877 8184 webserver.cc:460] Webserver started at http://127.0.97.66:39295/ using document root <none> and password file <none>
I20260620 15:17:06.915558 8184 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:06.915685 8184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:06.915995 8184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:06.918962 8184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "178453a42e9f433aafb200943b954a52"
format_stamp: "Formatted at 2026-06-20 15:17:06 on dist-test-slave-7s64"
server_key: "9617234d48c4dfafa2fdd7a6f3132526"
server_key_iv: "1f719ebfa82c8aa426447b061e0bef12"
server_key_version: "encryptionkey@0"
I20260620 15:17:06.919718 8184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "178453a42e9f433aafb200943b954a52"
format_stamp: "Formatted at 2026-06-20 15:17:06 on dist-test-slave-7s64"
server_key: "9617234d48c4dfafa2fdd7a6f3132526"
server_key_iv: "1f719ebfa82c8aa426447b061e0bef12"
server_key_version: "encryptionkey@0"
I20260620 15:17:06.924362 8184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 15:17:06.927531 8199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:06.928869 8184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:17:06.929025 8184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "178453a42e9f433aafb200943b954a52"
format_stamp: "Formatted at 2026-06-20 15:17:06 on dist-test-slave-7s64"
server_key: "9617234d48c4dfafa2fdd7a6f3132526"
server_key_iv: "1f719ebfa82c8aa426447b061e0bef12"
server_key_version: "encryptionkey@0"
I20260620 15:17:06.929232 8184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:06.949204 8184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:06.953384 8184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:06.953644 8184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:06.956140 8184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:06.956244 8184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:06.956382 8184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:06.956460 8184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:17:06 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968626, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:17:06.980441 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:06.956563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44599 (local address 127.0.97.126:38049)
0620 15:17:06.956712 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:06.956719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:06.958228 (+ 1509us) server_negotiation.cc:407] Connection header received
0620 15:17:06.960958 (+ 2730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:06.960964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:06.961042 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:06.961103 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:06.963264 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.964394 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.966859 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.967683 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.967847 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:06.973135 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:06.973141 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:06.973146 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:06.973187 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:06.976459 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:06.977461 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:06.977467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:06.977470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:06.977629 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:06.978292 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:06.978299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:06.978306 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:06.978719 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:06.978886 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:06.979895 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0620 15:17:06.980133 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 15:17:06.981774 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:06.957236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38049 (local address 127.0.97.66:44599)
0620 15:17:06.958070 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:06.958124 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:06.960646 (+ 2522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:06.961549 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:06.961559 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:06.962235 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:06.963057 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:06.963082 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.964615 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.964620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:06.966685 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:06.966701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.968245 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.968257 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:06.968633 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:06.969463 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:06.969507 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:06.972832 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:06.976652 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:06.976662 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:06.976678 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:06.977311 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:06.977760 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:06.977767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:06.977776 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:06.978122 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:06.978873 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:06.978886 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:06.979322 (+ 436us) client_negotiation.cc:769] Sending connection context
0620 15:17:06.979723 (+ 401us) client_negotiation.cc:240] Negotiation successful
0620 15:17:06.980204 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":336,"threads_started":1}
I20260620 15:17:07.003470 8266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:06.988447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38049 (local address 127.0.97.66:43357)
0620 15:17:06.989123 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:06.989138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:06.989241 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:06.989575 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:06.989578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:06.989995 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:06.990267 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:06.990275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.992050 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.992056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:06.993537 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:06.993548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.994445 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.994452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:06.994683 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:06.995421 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:06.995449 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:06.996628 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:07.000351 (+ 3723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:07.000357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:07.000360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:07.000867 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:07.001320 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:07.001326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:07.001331 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:07.001507 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:07.002173 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:07.002178 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:07.002428 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.002607 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.002937 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":361,"threads_started":1}
I20260620 15:17:07.004236 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:06.988450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:43357 (local address 127.0.97.126:38049)
0620 15:17:06.988688 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:06.988697 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:06.989213 (+ 516us) server_negotiation.cc:407] Connection header received
0620 15:17:06.989333 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:06.989341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:06.989436 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:06.989547 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:06.990399 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.991889 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.993679 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:06.994247 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:06.994390 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:06.996810 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:06.996820 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:06.996824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:06.996851 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:07.000227 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:07.001029 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:07.001034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:07.001038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:07.001183 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:07.001614 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:07.001619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:07.001623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:07.002058 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:07.002196 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.003793 (+ 1597us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.004024 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:17:07.014423 8184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39651
I20260620 15:17:07.014463 8318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39651 every 8 connection(s)
I20260620 15:17:07.015569 8184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:17:07.017627 8319 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38049
I20260620 15:17:07.017859 8319 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:07.017983 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 8184
I20260620 15:17:07.018172 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:17:07.018623 8319 heartbeater.cc:511] Master 127.0.97.126:38049 requested a full tablet report, sending...
I20260620 15:17:07.018721 389 external_mini_cluster.cc:1428] Setting key bc3d096762eef58588d7fd8cd9390f0c
I20260620 15:17:07.020107 7973 ts_manager.cc:194] Registered new tserver with Master: 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651)
I20260620 15:17:07.021062 7973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:44599
WARNING: no policy specified for oryx/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab.
I20260620 15:17:07.061691 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38049
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:07.379590 8323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:07.379853 8323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:07.379990 8323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:07.387271 8323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:07.387364 8323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:07.387526 8323 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:17:07.387638 8323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:17:07.397351 8323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:38049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:07.398459 8323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:07.399866 8323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:07.412037 8328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:07.412142 8331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:07.412142 8329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:07.413312 8323 server_base.cc:1034] running on GCE node
I20260620 15:17:07.413966 8323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:07.414927 8323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:07.416175 8323 hybrid_clock.cc:648] HybridClock initialized: now 1781968627416110 us; error 34 us; skew 500 ppm
I20260620 15:17:07.416476 8323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:07 dist-test-slave-7s64 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968627, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:07.419842 8323 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260620 15:17:07.421402 8323 webserver.cc:460] Webserver started at http://127.0.97.67:34203/ using document root <none> and password file <none>
I20260620 15:17:07.422075 8323 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:07.422210 8323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:07.422528 8323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:07.425479 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "ee235e52a7074d2a877bfbffac6cceae"
format_stamp: "Formatted at 2026-06-20 15:17:07 on dist-test-slave-7s64"
server_key: "9ed5769becf2ac04565ea26bc44db29c"
server_key_iv: "3ab69733de8f36cf6797651396dca04c"
server_key_version: "encryptionkey@0"
I20260620 15:17:07.426295 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "ee235e52a7074d2a877bfbffac6cceae"
format_stamp: "Formatted at 2026-06-20 15:17:07 on dist-test-slave-7s64"
server_key: "9ed5769becf2ac04565ea26bc44db29c"
server_key_iv: "3ab69733de8f36cf6797651396dca04c"
server_key_version: "encryptionkey@0"
I20260620 15:17:07.431381 8323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:17:07.435047 8338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:07.436281 8323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:17:07.436461 8323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "ee235e52a7074d2a877bfbffac6cceae"
format_stamp: "Formatted at 2026-06-20 15:17:07 on dist-test-slave-7s64"
server_key: "9ed5769becf2ac04565ea26bc44db29c"
server_key_iv: "3ab69733de8f36cf6797651396dca04c"
server_key_version: "encryptionkey@0"
I20260620 15:17:07.436733 8323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:07.450332 8323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:07.454175 8323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:07.454433 8323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:07.456940 8323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:07.457043 8323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:07.457191 8323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:07.457276 8323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:17:07 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968627, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:17:07.483525 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.458067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38049 (local address 127.0.97.67:59857)
0620 15:17:07.458979 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:07.459031 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:07.461796 (+ 2765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:07.462711 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:07.462725 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:07.463529 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:07.464300 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.464342 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.465968 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.465975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.467878 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.467889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.468955 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.468963 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.469278 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.470058 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:07.470115 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:07.473419 (+ 3304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:07.477349 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:07.477361 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:07.477392 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:07.478113 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:07.478552 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:07.478556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:07.478565 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:07.479272 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:07.479991 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:07.479999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:07.480565 (+ 566us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.480945 (+ 380us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.481602 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":381,"threads_started":1}
I20260620 15:17:07.484314 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.457520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59857 (local address 127.0.97.126:38049)
0620 15:17:07.457722 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.457728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.459172 (+ 1444us) server_negotiation.cc:407] Connection header received
0620 15:17:07.462213 (+ 3041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.462219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.462315 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.462426 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:07.464528 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.465795 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.468028 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.468691 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.468831 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.473739 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:07.473746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:07.473752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:07.473798 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:07.477185 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:07.478263 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:07.478268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:07.478270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:07.478404 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:07.479418 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:07.479426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:07.479431 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:07.479806 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:07.479974 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.483567 (+ 3593us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.483867 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 15:17:07.505939 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.490061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:42705 (local address 127.0.97.126:38049)
0620 15:17:07.490285 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.490294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.491010 (+ 716us) server_negotiation.cc:407] Connection header received
0620 15:17:07.491225 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.491233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.491340 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.491477 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:07.492689 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.493798 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.496053 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.496754 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.496924 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.499299 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:07.499305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:07.499309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:07.499338 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:07.502834 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:07.503756 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:07.503765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:07.503770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:07.503949 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:07.504385 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:07.504392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:07.504399 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:07.504721 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:07.504890 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.505307 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.505528 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 15:17:07.506063 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.490082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38049 (local address 127.0.97.67:42705)
0620 15:17:07.490883 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:07.490906 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:07.491065 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:07.491630 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:07.491633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:07.492127 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:07.492522 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.492537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.493950 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.493958 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.495831 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.495852 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.497193 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.497199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.497358 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.498182 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:07.498209 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:07.499114 (+ 905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:07.502975 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:07.502982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:07.502986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:07.503631 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:07.504086 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:07.504093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:07.504096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:07.504281 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:07.504843 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:07.504849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:07.505045 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.505201 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.505536 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":438,"threads_started":1}
I20260620 15:17:07.510723 8323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:32921
I20260620 15:17:07.510746 8457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:32921 every 8 connection(s)
I20260620 15:17:07.511904 8323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:17:07.514225 8458 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38049
I20260620 15:17:07.514516 8458 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:07.515451 8458 heartbeater.cc:511] Master 127.0.97.126:38049 requested a full tablet report, sending...
I20260620 15:17:07.517144 7973 ts_manager.cc:194] Registered new tserver with Master: ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921)
I20260620 15:17:07.518054 7973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:59857
I20260620 15:17:07.519977 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 8323
I20260620 15:17:07.520166 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:17:07.520617 389 external_mini_cluster.cc:1428] Setting key b4ff5cb1c6d8862e7c748841ee6798b6
I20260620 15:17:07.523190 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:17:07.523669 8180 heartbeater.cc:503] Master 127.0.97.126:38049 was elected leader, sending a full tablet report...
Jun 20 15:17:07 dist-test-slave-7s64 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968627, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:17:07 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:07 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:17:07.569103 8022 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:07.559494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.0.97.126:38049)
0620 15:17:07.559713 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.559719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.559873 (+ 154us) server_negotiation.cc:407] Connection header received
0620 15:17:07.560054 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.560059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.560123 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.560212 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:07.561472 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.562179 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.563311 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.563730 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.563828 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.568799 (+ 4971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55776: BlockingRecv error: recv got EOF from 127.0.0.1:55776 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 15:17:07 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968627, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:17:07.583810 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.571771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.0.97.126:38049)
0620 15:17:07.571918 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.571922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.572273 (+ 351us) server_negotiation.cc:407] Connection header received
0620 15:17:07.572373 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.572377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.572428 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.572601 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:07.573628 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.574291 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.575401 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.575781 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.575867 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.579167 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:07.579172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:07.579176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:07.579202 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:07.581659 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:07.582302 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:07.582306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:07.582308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:07.582412 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:07.582724 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:07.582727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:07.582729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:07.582957 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:07.583182 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.583380 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.583523 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:17:07.587780 7973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:17:07.597692 7973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:17:07.626284 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.611068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.0.97.67:32921)
0620 15:17:07.617435 (+ 6367us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.617443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.617476 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:17:07.617606 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.617613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.617825 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.618047 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:07.618831 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.620998 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.623776 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.625187 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.625328 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.625560 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.625799 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.625946 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6199,"thread_start_us":6041,"threads_started":1}
I20260620 15:17:07.626367 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.610726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:32921 (local address 127.0.0.1:37170)
0620 15:17:07.613357 (+ 2631us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:07.613394 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:07.613554 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:07.618051 (+ 4497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:07.618060 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:07.618105 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:07.618595 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.618614 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.621126 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.621132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.623611 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.623627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.625357 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.625362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.625554 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.625615 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.625753 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.625943 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2363,"mutex_wait_us":1,"thread_start_us":2556,"threads_started":1}
I20260620 15:17:07.626860 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.610988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37463 (local address 127.0.0.1:47500)
0620 15:17:07.618743 (+ 7755us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:07.618760 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:07.618864 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:07.619571 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:07.619577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:07.619601 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:07.620141 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.620153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.622008 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.622017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.624088 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.624101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.626081 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.626087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.626248 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.626279 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.626378 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.626516 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7616,"thread_start_us":7446,"threads_started":1}
I20260620 15:17:07.627516 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.611463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.0.97.65:37463)
0620 15:17:07.616916 (+ 5453us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.616924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.618828 (+ 1904us) server_negotiation.cc:407] Connection header received
0620 15:17:07.619019 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.619023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.619174 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.619370 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:07.620323 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.621857 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.624259 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.625624 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.625719 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.625892 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.627087 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.627239 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5296,"thread_start_us":5168,"threads_started":1}
I20260620 15:17:07.627580 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.619072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.0.97.66:39651)
0620 15:17:07.619599 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.619607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.620322 (+ 715us) server_negotiation.cc:407] Connection header received
0620 15:17:07.620998 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.621005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.621211 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.621376 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:07.621950 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.622826 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.625080 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.626394 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.626496 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.626671 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.627180 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.627311 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":248,"threads_started":1}
I20260620 15:17:07.628068 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.619606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39651 (local address 127.0.0.1:33518)
0620 15:17:07.620236 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:07.620252 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:07.620423 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:07.621367 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:07.621371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:07.621407 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:07.621795 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.621803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.622967 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.622972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.624927 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.624941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.626781 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.626788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.626992 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.627027 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.627716 (+ 689us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.627811 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":1,"thread_start_us":329,"threads_started":1}
I20260620 15:17:07.630170 8388 tablet_service.cc:1467] Processing CreateTablet for tablet 624d3e7b812f4bb3b36644dca054cc0e (DEFAULT_TABLE table=test-table [id=1aead907bc29470e8f596bb51caacded]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:07.631227 8249 tablet_service.cc:1467] Processing CreateTablet for tablet 624d3e7b812f4bb3b36644dca054cc0e (DEFAULT_TABLE table=test-table [id=1aead907bc29470e8f596bb51caacded]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:07.631655 8388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 624d3e7b812f4bb3b36644dca054cc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:07.632125 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 624d3e7b812f4bb3b36644dca054cc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:07.632370 8110 tablet_service.cc:1467] Processing CreateTablet for tablet 624d3e7b812f4bb3b36644dca054cc0e (DEFAULT_TABLE table=test-table [id=1aead907bc29470e8f596bb51caacded]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:07.633576 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 624d3e7b812f4bb3b36644dca054cc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:07.642526 8482 tablet_bootstrap.cc:492] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: Bootstrap starting.
I20260620 15:17:07.646082 8482 tablet_bootstrap.cc:654] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:07.646540 8483 tablet_bootstrap.cc:492] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: Bootstrap starting.
I20260620 15:17:07.647392 8482 log.cc:824] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:07.647877 8484 tablet_bootstrap.cc:492] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: Bootstrap starting.
I20260620 15:17:07.650557 8484 tablet_bootstrap.cc:654] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:07.650584 8483 tablet_bootstrap.cc:654] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:07.651517 8484 log.cc:824] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:07.651954 8483 log.cc:824] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:07.654189 8484 tablet_bootstrap.cc:492] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: No bootstrap required, opened a new log
I20260620 15:17:07.654436 8484 ts_tablet_manager.cc:1397] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 15:17:07.658457 8482 tablet_bootstrap.cc:492] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: No bootstrap required, opened a new log
I20260620 15:17:07.658718 8482 ts_tablet_manager.cc:1397] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20260620 15:17:07.658979 8483 tablet_bootstrap.cc:492] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: No bootstrap required, opened a new log
I20260620 15:17:07.659212 8483 ts_tablet_manager.cc:1397] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260620 15:17:07.663724 8484 raft_consensus.cc:348] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } }
I20260620 15:17:07.664355 8484 raft_consensus.cc:374] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:07.664536 8484 raft_consensus.cc:729] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a97c74af9844d28027d062ec3d7851, State: Initialized, Role: FOLLOWER
I20260620 15:17:07.665236 8484 consensus_queue.cc:260] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } }
I20260620 15:17:07.665674 8482 raft_consensus.cc:348] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } }
I20260620 15:17:07.666059 8482 raft_consensus.cc:374] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:07.666195 8482 raft_consensus.cc:729] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178453a42e9f433aafb200943b954a52, State: Initialized, Role: FOLLOWER
I20260620 15:17:07.666838 8482 consensus_queue.cc:260] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } }
I20260620 15:17:07.667057 8484 ts_tablet_manager.cc:1428] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 15:17:07.668840 8319 heartbeater.cc:503] Master 127.0.97.126:38049 was elected leader, sending a full tablet report...
I20260620 15:17:07.669138 8483 raft_consensus.cc:348] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } }
I20260620 15:17:07.669422 8482 ts_tablet_manager.cc:1428] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260620 15:17:07.669512 8483 raft_consensus.cc:374] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:07.669600 8483 raft_consensus.cc:729] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee235e52a7074d2a877bfbffac6cceae, State: Initialized, Role: FOLLOWER
I20260620 15:17:07.670137 8483 consensus_queue.cc:260] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } }
I20260620 15:17:07.671218 8458 heartbeater.cc:503] Master 127.0.97.126:38049 was elected leader, sending a full tablet report...
I20260620 15:17:07.671584 8483 ts_tablet_manager.cc:1428] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 15:17:07.703321 8489 raft_consensus.cc:482] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:17:07.703574 8489 raft_consensus.cc:504] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } }
I20260620 15:17:07.705274 8489 leader_election.cc:288] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921), a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463)
I20260620 15:17:07.712052 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.705849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:60915 (local address 127.0.97.65:37463)
0620 15:17:07.706038 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.706047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.706780 (+ 733us) server_negotiation.cc:407] Connection header received
0620 15:17:07.706975 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.706979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.707041 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.707134 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:07.707715 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.708384 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.710244 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.711027 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.711168 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.711266 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.711664 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.711793 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:17:07.712210 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.705728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37463 (local address 127.0.97.66:60915)
0620 15:17:07.706676 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:07.706694 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:07.706841 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:07.707150 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:07.707154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:07.707191 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:07.707571 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.707580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.708555 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.708562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.710093 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.710104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.711261 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.711267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.711444 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.711488 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.711640 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.711779 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":55,"thread_start_us":644,"threads_started":1}
I20260620 15:17:07.712226 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.705515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:35177 (local address 127.0.97.67:32921)
0620 15:17:07.705733 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.705744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.706576 (+ 832us) server_negotiation.cc:407] Connection header received
0620 15:17:07.706811 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.706817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.706879 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.706975 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:07.707596 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.708315 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.710269 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.711246 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.711388 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.711496 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.711897 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.712006 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:17:07.712824 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.705485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:32921 (local address 127.0.97.66:35177)
0620 15:17:07.706456 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:07.706474 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:07.706653 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:07.707002 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:07.707007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:07.707034 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:07.707433 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.707444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.708458 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.708465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.710110 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.710122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.711530 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.711536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.711716 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.711750 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.712418 (+ 668us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.712576 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":6,"thread_start_us":491,"threads_started":1}
I20260620 15:17:07.713404 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624d3e7b812f4bb3b36644dca054cc0e" candidate_uuid: "178453a42e9f433aafb200943b954a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a97c74af9844d28027d062ec3d7851" is_pre_election: true
I20260620 15:17:07.713841 8130 raft_consensus.cc:2454] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 178453a42e9f433aafb200943b954a52 in term 0.
I20260620 15:17:07.713758 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624d3e7b812f4bb3b36644dca054cc0e" candidate_uuid: "178453a42e9f433aafb200943b954a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee235e52a7074d2a877bfbffac6cceae" is_pre_election: true
I20260620 15:17:07.714259 8408 raft_consensus.cc:2454] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 178453a42e9f433aafb200943b954a52 in term 0.
I20260620 15:17:07.714440 8201 leader_election.cc:302] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178453a42e9f433aafb200943b954a52, a8a97c74af9844d28027d062ec3d7851; no voters:
I20260620 15:17:07.714846 8489 raft_consensus.cc:2781] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:17:07.714984 8489 raft_consensus.cc:482] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:17:07.715112 8489 raft_consensus.cc:3037] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:07.717423 8489 raft_consensus.cc:504] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } }
I20260620 15:17:07.718032 8489 leader_election.cc:288] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [CANDIDATE]: Term 1 election: Requested vote from peers ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921), a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463)
I20260620 15:17:07.718454 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624d3e7b812f4bb3b36644dca054cc0e" candidate_uuid: "178453a42e9f433aafb200943b954a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee235e52a7074d2a877bfbffac6cceae"
I20260620 15:17:07.718524 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624d3e7b812f4bb3b36644dca054cc0e" candidate_uuid: "178453a42e9f433aafb200943b954a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a97c74af9844d28027d062ec3d7851"
I20260620 15:17:07.718634 8408 raft_consensus.cc:3037] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:07.718724 8130 raft_consensus.cc:3037] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:07.720875 8130 raft_consensus.cc:2454] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 178453a42e9f433aafb200943b954a52 in term 1.
I20260620 15:17:07.720901 8408 raft_consensus.cc:2454] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 178453a42e9f433aafb200943b954a52 in term 1.
I20260620 15:17:07.721272 8200 leader_election.cc:302] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178453a42e9f433aafb200943b954a52, ee235e52a7074d2a877bfbffac6cceae; no voters:
I20260620 15:17:07.721622 8489 raft_consensus.cc:2781] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:07.722038 8489 raft_consensus.cc:686] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 1 LEADER]: Becoming Leader. State: Replica: 178453a42e9f433aafb200943b954a52, State: Running, Role: LEADER
I20260620 15:17:07.722460 8489 consensus_queue.cc:237] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } }
I20260620 15:17:07.726781 7972 catalog_manager.cc:5495] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 178453a42e9f433aafb200943b954a52 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "178453a42e9f433aafb200943b954a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } health_report { overall_health: HEALTHY } } }
I20260620 15:17:07.746285 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.741491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.0.97.66:39651)
0620 15:17:07.741691 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.741698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.741807 (+ 109us) server_negotiation.cc:407] Connection header received
0620 15:17:07.741953 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.741957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.742022 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.742132 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:17:07.742416 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.743106 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.744276 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.744719 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.744807 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.745248 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:17:07.745734 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:17:07.745837 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.745944 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.746000 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:17:07.753726 8130 raft_consensus.cc:1261] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Refusing update from remote peer 178453a42e9f433aafb200943b954a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:07.753729 8408 raft_consensus.cc:1261] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Refusing update from remote peer 178453a42e9f433aafb200943b954a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:07.754663 8493 consensus_queue.cc:1035] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:07.755028 8489 consensus_queue.cc:1035] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:07.765386 8498 mvcc.cc:205] Tried to move back new op lower bound from 7298943499269996544 to 7298943499154522112. Current Snapshot: MvccSnapshot[applied={T|T < 7298943499269996544}]
W20260620 15:17:07.766824 8181 tablet.cc:2369] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:17:07.780478 7974 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:07.780756 7974 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:07.783831 7974 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da: Sending DeleteTablet for 3 replicas of tablet 624d3e7b812f4bb3b36644dca054cc0e
I20260620 15:17:07.784807 8388 tablet_service.cc:1514] Processing DeleteTablet for tablet 624d3e7b812f4bb3b36644dca054cc0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:07 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:37170
I20260620 15:17:07.785006 8110 tablet_service.cc:1514] Processing DeleteTablet for tablet 624d3e7b812f4bb3b36644dca054cc0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:07 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:47500
I20260620 15:17:07.785189 8249 tablet_service.cc:1514] Processing DeleteTablet for tablet 624d3e7b812f4bb3b36644dca054cc0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:07 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:33518
I20260620 15:17:07.787254 7974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:17:07.788015 7974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:17:07.798825 8248 tablet_service.cc:1467] Processing CreateTablet for tablet 9a71c61331ad415daaf31b4d1e62fbfb (DEFAULT_TABLE table=test-table [id=7b8ef2e20f3c4dc697c31963f23d0815]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:07.798846 8109 tablet_service.cc:1467] Processing CreateTablet for tablet 9a71c61331ad415daaf31b4d1e62fbfb (DEFAULT_TABLE table=test-table [id=7b8ef2e20f3c4dc697c31963f23d0815]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:07.799468 8109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a71c61331ad415daaf31b4d1e62fbfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:07.799494 8248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a71c61331ad415daaf31b4d1e62fbfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:07.800060 8505 tablet_replica.cc:330] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: stopping tablet replica
I20260620 15:17:07.800585 8505 raft_consensus.cc:2229] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:07.801199 8505 raft_consensus.cc:2258] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:07.803176 8505 ts_tablet_manager.cc:1905] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:07.805222 8388 tablet_service.cc:1467] Processing CreateTablet for tablet 9a71c61331ad415daaf31b4d1e62fbfb (DEFAULT_TABLE table=test-table [id=7b8ef2e20f3c4dc697c31963f23d0815]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:07.805668 8388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a71c61331ad415daaf31b4d1e62fbfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:07.808212 8482 tablet_bootstrap.cc:492] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52: Bootstrap starting.
I20260620 15:17:07.810215 8482 tablet_bootstrap.cc:654] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:07.812912 8484 tablet_bootstrap.cc:492] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851: Bootstrap starting.
I20260620 15:17:07.814407 8484 tablet_bootstrap.cc:654] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:07.816526 8483 tablet_bootstrap.cc:492] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae: Bootstrap starting.
I20260620 15:17:07.818567 8483 tablet_bootstrap.cc:654] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:07.819722 8507 tablet_replica.cc:330] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: stopping tablet replica
I20260620 15:17:07.820775 8507 raft_consensus.cc:2229] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:17:07.823004 8507 raft_consensus.cc:2258] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:07.823843 8482 tablet_bootstrap.cc:492] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52: No bootstrap required, opened a new log
I20260620 15:17:07.824015 8482 ts_tablet_manager.cc:1397] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260620 15:17:07.825337 8482 raft_consensus.cc:348] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.825578 8482 raft_consensus.cc:374] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:07.825599 8505 ts_tablet_manager.cc:1918] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:07.825759 8505 log.cc:1196] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/624d3e7b812f4bb3b36644dca054cc0e
I20260620 15:17:07.825730 8482 raft_consensus.cc:729] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178453a42e9f433aafb200943b954a52, State: Initialized, Role: FOLLOWER
I20260620 15:17:07.825990 8482 consensus_queue.cc:260] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.826655 8482 ts_tablet_manager.cc:1428] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:17:07.827736 8506 tablet_replica.cc:330] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: stopping tablet replica
I20260620 15:17:07.825580 8507 ts_tablet_manager.cc:1905] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:07.831118 8505 ts_tablet_manager.cc:1939] T 624d3e7b812f4bb3b36644dca054cc0e P ee235e52a7074d2a877bfbffac6cceae: Deleting consensus metadata
I20260620 15:17:07.832598 8506 raft_consensus.cc:2229] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:07.833146 8506 raft_consensus.cc:2258] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:07.834169 8483 tablet_bootstrap.cc:492] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae: No bootstrap required, opened a new log
I20260620 15:17:07.834378 8483 ts_tablet_manager.cc:1397] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.011s
I20260620 15:17:07.834553 7958 catalog_manager.cc:4841] TS ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921): tablet 624d3e7b812f4bb3b36644dca054cc0e (table test-table [id=1aead907bc29470e8f596bb51caacded]) successfully deleted
I20260620 15:17:07.835222 8483 raft_consensus.cc:348] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.835384 8506 ts_tablet_manager.cc:1905] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:07.835440 8483 raft_consensus.cc:374] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:07.835527 8483 raft_consensus.cc:729] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee235e52a7074d2a877bfbffac6cceae, State: Initialized, Role: FOLLOWER
I20260620 15:17:07.835778 8483 consensus_queue.cc:260] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.836454 8483 ts_tablet_manager.cc:1428] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 15:17:07.837608 8484 tablet_bootstrap.cc:492] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851: No bootstrap required, opened a new log
I20260620 15:17:07.837795 8484 ts_tablet_manager.cc:1397] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.015s
I20260620 15:17:07.838682 8484 raft_consensus.cc:348] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.840569 8484 raft_consensus.cc:374] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:07.840745 8484 raft_consensus.cc:729] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a97c74af9844d28027d062ec3d7851, State: Initialized, Role: FOLLOWER
I20260620 15:17:07.841147 8484 consensus_queue.cc:260] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.841727 8484 ts_tablet_manager.cc:1428] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 15:17:07.843240 8507 ts_tablet_manager.cc:1918] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:07.843410 8507 log.cc:1196] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/624d3e7b812f4bb3b36644dca054cc0e
I20260620 15:17:07.843941 8507 ts_tablet_manager.cc:1939] T 624d3e7b812f4bb3b36644dca054cc0e P 178453a42e9f433aafb200943b954a52: Deleting consensus metadata
I20260620 15:17:07.842636 8506 ts_tablet_manager.cc:1918] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:07.844762 8506 log.cc:1196] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/624d3e7b812f4bb3b36644dca054cc0e
I20260620 15:17:07.845417 8506 ts_tablet_manager.cc:1939] T 624d3e7b812f4bb3b36644dca054cc0e P a8a97c74af9844d28027d062ec3d7851: Deleting consensus metadata
I20260620 15:17:07.845566 7959 catalog_manager.cc:4841] TS 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651): tablet 624d3e7b812f4bb3b36644dca054cc0e (table test-table [id=1aead907bc29470e8f596bb51caacded]) successfully deleted
I20260620 15:17:07.847057 7959 catalog_manager.cc:4841] TS a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463): tablet 624d3e7b812f4bb3b36644dca054cc0e (table test-table [id=1aead907bc29470e8f596bb51caacded]) successfully deleted
I20260620 15:17:07.858668 8488 raft_consensus.cc:482] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:17:07.858878 8488 raft_consensus.cc:504] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.860673 8488 leader_election.cc:288] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651), ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921)
I20260620 15:17:07.868539 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.861093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:32921 (local address 127.0.97.65:41613)
0620 15:17:07.862474 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:07.862491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:07.862644 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:07.862973 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:07.862977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:07.862999 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:07.863311 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.863318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.864308 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.864315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.866280 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.866291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.867648 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.867655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.867830 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.867864 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.867966 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.868094 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"mutex_wait_us":349,"spinlock_wait_cycles":5888,"thread_start_us":591,"threads_started":1}
I20260620 15:17:07.868810 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.861520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:41613 (local address 127.0.97.67:32921)
0620 15:17:07.861685 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.861691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.862576 (+ 885us) server_negotiation.cc:407] Connection header received
0620 15:17:07.862784 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.862788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.862854 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.862947 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:07.863472 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.864188 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.866466 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.867432 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.867563 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.867654 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.868004 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.868130 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:17:07.869518 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.861013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39651 (local address 127.0.97.65:53023)
0620 15:17:07.861726 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:07.861750 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:07.861902 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:07.862435 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:07.862439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:07.862469 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:07.862843 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.862855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.864383 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.864388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.866795 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.866811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.868554 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.868561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.868863 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.868908 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.869017 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.869150 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":3,"spinlock_wait_cycles":29568,"thread_start_us":1168,"threads_started":1}
I20260620 15:17:07.869560 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.861262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:53023 (local address 127.0.97.66:39651)
0620 15:17:07.861428 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.861436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.861823 (+ 387us) server_negotiation.cc:407] Connection header received
0620 15:17:07.862092 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.862098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.862173 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.862273 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:07.863031 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.864204 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.866993 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.868131 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.868271 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.868376 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.869058 (+ 682us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.869196 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 15:17:07.869544 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a71c61331ad415daaf31b4d1e62fbfb" candidate_uuid: "a8a97c74af9844d28027d062ec3d7851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee235e52a7074d2a877bfbffac6cceae" is_pre_election: true
I20260620 15:17:07.869794 8408 raft_consensus.cc:2454] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a97c74af9844d28027d062ec3d7851 in term 0.
I20260620 15:17:07.870267 8061 leader_election.cc:302] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8a97c74af9844d28027d062ec3d7851, ee235e52a7074d2a877bfbffac6cceae; no voters:
I20260620 15:17:07.870656 8488 raft_consensus.cc:2781] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:17:07.870774 8488 raft_consensus.cc:482] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:17:07.870860 8488 raft_consensus.cc:3037] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:07.870666 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a71c61331ad415daaf31b4d1e62fbfb" candidate_uuid: "a8a97c74af9844d28027d062ec3d7851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178453a42e9f433aafb200943b954a52" is_pre_election: true
I20260620 15:17:07.871134 8274 raft_consensus.cc:2454] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a97c74af9844d28027d062ec3d7851 in term 0.
I20260620 15:17:07.873946 8488 raft_consensus.cc:504] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.874614 8488 leader_election.cc:288] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [CANDIDATE]: Term 1 election: Requested vote from peers 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651), ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921)
I20260620 15:17:07.874923 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a71c61331ad415daaf31b4d1e62fbfb" candidate_uuid: "a8a97c74af9844d28027d062ec3d7851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178453a42e9f433aafb200943b954a52"
I20260620 15:17:07.875159 8274 raft_consensus.cc:3037] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:07.875301 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a71c61331ad415daaf31b4d1e62fbfb" candidate_uuid: "a8a97c74af9844d28027d062ec3d7851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee235e52a7074d2a877bfbffac6cceae"
I20260620 15:17:07.875510 8408 raft_consensus.cc:3037] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:07.878379 8408 raft_consensus.cc:2454] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a97c74af9844d28027d062ec3d7851 in term 1.
I20260620 15:17:07.878386 8274 raft_consensus.cc:2454] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a97c74af9844d28027d062ec3d7851 in term 1.
I20260620 15:17:07.878731 8062 leader_election.cc:302] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178453a42e9f433aafb200943b954a52, a8a97c74af9844d28027d062ec3d7851; no voters:
I20260620 15:17:07.879041 8488 raft_consensus.cc:2781] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:07.879453 8488 raft_consensus.cc:686] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 1 LEADER]: Becoming Leader. State: Replica: a8a97c74af9844d28027d062ec3d7851, State: Running, Role: LEADER
I20260620 15:17:07.879859 8488 consensus_queue.cc:237] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.883396 7974 catalog_manager.cc:5495] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8a97c74af9844d28027d062ec3d7851 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "a8a97c74af9844d28027d062ec3d7851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } health_report { overall_health: UNKNOWN } } }
Jun 20 15:17:07 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968625, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:17:07.933240 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.918795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38049 (local address 127.0.0.1:55784)
0620 15:17:07.919439 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:07.919458 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:07.919569 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:07.919938 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:07.919941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:07.920644 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:07.920906 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.920914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.922270 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.922274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.923204 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:07.923213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.924156 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.924160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:07.924283 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.924942 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:07.924964 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:07.926923 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:07.930518 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:07.930528 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:07.930534 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:07.930986 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:07.931359 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:07.931363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:07.931365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:07.931508 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:07.932146 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:07.932172 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:07.932342 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 15:17:07.932552 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 15:17:07.932858 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":371,"threads_started":1}
I20260620 15:17:07.933652 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:07.918779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.0.97.126:38049)
0620 15:17:07.918950 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:07.918956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:07.919541 (+ 585us) server_negotiation.cc:407] Connection header received
0620 15:17:07.919693 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:07.919699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:07.919794 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:07.919896 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:07.921045 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.922145 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.923328 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:07.923984 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:07.924117 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:07.927098 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:07.927104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:07.927110 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:07.927146 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:07.930375 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:07.931103 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:07.931108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:07.931112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:07.931249 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:07.931660 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:07.931666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:07.931669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:07.932003 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:07.932163 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:07.932698 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 15:17:07.933046 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":41}
I20260620 15:17:07.938210 7974 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:55784:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:17:07.939136 7974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:17:07.945741 8110 tablet_service.cc:1467] Processing CreateTablet for tablet 39e6fe184218422fbdb7ab93595384d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41706ae99d4947d880ad3a3382a1b28b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:17:07.945792 8249 tablet_service.cc:1467] Processing CreateTablet for tablet 39e6fe184218422fbdb7ab93595384d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41706ae99d4947d880ad3a3382a1b28b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:17:07.946009 8388 tablet_service.cc:1467] Processing CreateTablet for tablet 39e6fe184218422fbdb7ab93595384d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41706ae99d4947d880ad3a3382a1b28b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:17:07.946275 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e6fe184218422fbdb7ab93595384d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:07.946329 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e6fe184218422fbdb7ab93595384d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:07.946471 8388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e6fe184218422fbdb7ab93595384d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:07.953641 8483 tablet_bootstrap.cc:492] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: Bootstrap starting.
I20260620 15:17:07.955224 8483 tablet_bootstrap.cc:654] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:07.956184 8484 tablet_bootstrap.cc:492] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: Bootstrap starting.
I20260620 15:17:07.957306 8482 tablet_bootstrap.cc:492] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: Bootstrap starting.
I20260620 15:17:07.957828 8484 tablet_bootstrap.cc:654] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:07.958849 8482 tablet_bootstrap.cc:654] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:07.960350 8483 tablet_bootstrap.cc:492] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: No bootstrap required, opened a new log
I20260620 15:17:07.960578 8483 ts_tablet_manager.cc:1397] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260620 15:17:07.961225 8483 raft_consensus.cc:348] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.961423 8483 raft_consensus.cc:374] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:07.961558 8483 raft_consensus.cc:729] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee235e52a7074d2a877bfbffac6cceae, State: Initialized, Role: FOLLOWER
I20260620 15:17:07.961865 8483 consensus_queue.cc:260] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.962461 8483 ts_tablet_manager.cc:1428] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 15:17:07.962332 8490 tablet_replica.cc:439] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } }
I20260620 15:17:07.962774 8490 tablet_replica.cc:442] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:17:07.969610 8484 tablet_bootstrap.cc:492] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: No bootstrap required, opened a new log
I20260620 15:17:07.969763 8484 ts_tablet_manager.cc:1397] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260620 15:17:07.970428 8484 raft_consensus.cc:348] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.970661 8484 raft_consensus.cc:374] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:07.970741 8484 raft_consensus.cc:729] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a97c74af9844d28027d062ec3d7851, State: Initialized, Role: FOLLOWER
I20260620 15:17:07.971024 8484 consensus_queue.cc:260] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.971412 8488 tablet_replica.cc:439] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } }
I20260620 15:17:07.971686 8488 tablet_replica.cc:442] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:17:07.974372 8484 ts_tablet_manager.cc:1428] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 15:17:07.974794 8482 tablet_bootstrap.cc:492] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: No bootstrap required, opened a new log
I20260620 15:17:07.974977 8482 ts_tablet_manager.cc:1397] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.007s
I20260620 15:17:07.975725 8482 raft_consensus.cc:348] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.976047 8482 raft_consensus.cc:374] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:07.976161 8482 raft_consensus.cc:729] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178453a42e9f433aafb200943b954a52, State: Initialized, Role: FOLLOWER
I20260620 15:17:07.976414 8482 consensus_queue.cc:260] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:07.976987 8482 ts_tablet_manager.cc:1428] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 15:17:07.976866 8493 tablet_replica.cc:439] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } }
I20260620 15:17:07.977183 8493 tablet_replica.cc:442] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 15:17:08.018342 8459 tablet.cc:2369] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:17:08.019271 8320 tablet.cc:2369] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:17:08.113104 8493 raft_consensus.cc:482] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:17:08.113363 8493 raft_consensus.cc:504] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.114032 8493 leader_election.cc:288] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463), ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921)
I20260620 15:17:08.114512 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e6fe184218422fbdb7ab93595384d0" candidate_uuid: "178453a42e9f433aafb200943b954a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a97c74af9844d28027d062ec3d7851" is_pre_election: true
I20260620 15:17:08.114627 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e6fe184218422fbdb7ab93595384d0" candidate_uuid: "178453a42e9f433aafb200943b954a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee235e52a7074d2a877bfbffac6cceae" is_pre_election: true
I20260620 15:17:08.114776 8130 raft_consensus.cc:2454] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 178453a42e9f433aafb200943b954a52 in term 0.
I20260620 15:17:08.114851 8408 raft_consensus.cc:2454] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 178453a42e9f433aafb200943b954a52 in term 0.
I20260620 15:17:08.115187 8201 leader_election.cc:302] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178453a42e9f433aafb200943b954a52, a8a97c74af9844d28027d062ec3d7851; no voters:
I20260620 15:17:08.115469 8493 raft_consensus.cc:2781] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:17:08.115625 8493 raft_consensus.cc:482] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:17:08.115762 8493 raft_consensus.cc:3037] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:08.118268 8493 raft_consensus.cc:504] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.119122 8493 leader_election.cc:288] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [CANDIDATE]: Term 1 election: Requested vote from peers a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463), ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921)
I20260620 15:17:08.119222 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e6fe184218422fbdb7ab93595384d0" candidate_uuid: "178453a42e9f433aafb200943b954a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a97c74af9844d28027d062ec3d7851"
I20260620 15:17:08.119421 8130 raft_consensus.cc:3037] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:08.119413 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e6fe184218422fbdb7ab93595384d0" candidate_uuid: "178453a42e9f433aafb200943b954a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee235e52a7074d2a877bfbffac6cceae"
I20260620 15:17:08.119639 8408 raft_consensus.cc:3037] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:08.122342 8408 raft_consensus.cc:2454] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 178453a42e9f433aafb200943b954a52 in term 1.
I20260620 15:17:08.122357 8130 raft_consensus.cc:2454] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 178453a42e9f433aafb200943b954a52 in term 1.
I20260620 15:17:08.122736 8201 leader_election.cc:302] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178453a42e9f433aafb200943b954a52, a8a97c74af9844d28027d062ec3d7851; no voters:
I20260620 15:17:08.122982 8493 raft_consensus.cc:2781] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:08.123162 8493 raft_consensus.cc:686] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [term 1 LEADER]: Becoming Leader. State: Replica: 178453a42e9f433aafb200943b954a52, State: Running, Role: LEADER
I20260620 15:17:08.123420 8493 consensus_queue.cc:237] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.125161 8489 tablet_replica.cc:439] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: TxnStatusTablet state changed. Reason: New leader 178453a42e9f433aafb200943b954a52. Latest consensus state: current_term: 1 leader_uuid: "178453a42e9f433aafb200943b954a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } }
I20260620 15:17:08.125438 8489 tablet_replica.cc:442] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:17:08.126125 8525 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:17:08.126304 8525 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:17:08.126837 7974 catalog_manager.cc:5495] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 178453a42e9f433aafb200943b954a52 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "178453a42e9f433aafb200943b954a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } health_report { overall_health: UNKNOWN } } }
I20260620 15:17:08.157697 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.152292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.0.97.66:39651)
0620 15:17:08.152453 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:08.152457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:08.152474 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:17:08.152802 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:08.152806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:08.152878 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:08.152962 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:17:08.153345 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.154105 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.155957 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.156418 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.156610 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.157093 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:17:08.157181 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:17:08.157247 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:08.157420 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 15:17:08.157489 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:17:08.157843 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.152167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39651 (local address 127.0.0.1:33526)
0620 15:17:08.152407 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:08.152428 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:08.152597 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:08.153011 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:08.153016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:08.153030 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:17:08.153217 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:08.153227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.154239 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.154246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:08.155808 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:08.155824 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.156659 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.156665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:08.156848 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.156929 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:17:08.157281 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:17:08.157306 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:17:08.157403 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 15:17:08.157551 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 15:17:08.161067 8408 raft_consensus.cc:1261] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Refusing update from remote peer 178453a42e9f433aafb200943b954a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:08.161087 8130 raft_consensus.cc:1261] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Refusing update from remote peer 178453a42e9f433aafb200943b954a52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:08.161643 8489 consensus_queue.cc:1035] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:08.161998 8493 consensus_queue.cc:1035] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:08.164783 8490 tablet_replica.cc:439] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: TxnStatusTablet state changed. Reason: New leader 178453a42e9f433aafb200943b954a52. Latest consensus state: current_term: 1 leader_uuid: "178453a42e9f433aafb200943b954a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } }
I20260620 15:17:08.165040 8490 tablet_replica.cc:442] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:17:08.166338 8493 tablet_replica.cc:439] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "178453a42e9f433aafb200943b954a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } }
I20260620 15:17:08.166610 8493 tablet_replica.cc:442] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:17:08.167086 8496 mvcc.cc:205] Tried to move back new op lower bound from 7298943500942077952 to 7298943500796076032. Current Snapshot: MvccSnapshot[applied={T|T < 7298943500942077952 or (T in {7298943500942077952})}]
I20260620 15:17:08.168053 8488 tablet_replica.cc:439] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: TxnStatusTablet state changed. Reason: New leader 178453a42e9f433aafb200943b954a52. Latest consensus state: current_term: 1 leader_uuid: "178453a42e9f433aafb200943b954a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } }
I20260620 15:17:08.168344 8488 tablet_replica.cc:442] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:17:08.169888 8490 tablet_replica.cc:439] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "178453a42e9f433aafb200943b954a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } }
I20260620 15:17:08.170118 8490 tablet_replica.cc:442] T 39e6fe184218422fbdb7ab93595384d0 P ee235e52a7074d2a877bfbffac6cceae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:17:08.172159 8489 tablet_replica.cc:439] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "178453a42e9f433aafb200943b954a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } }
I20260620 15:17:08.172416 8489 tablet_replica.cc:442] T 39e6fe184218422fbdb7ab93595384d0 P 178453a42e9f433aafb200943b954a52: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:17:08.174026 8497 mvcc.cc:205] Tried to move back new op lower bound from 7298943500942077952 to 7298943500796076032. Current Snapshot: MvccSnapshot[applied={T|T < 7298943500942077952 or (T in {7298943500942077952})}]
I20260620 15:17:08.175568 8488 tablet_replica.cc:439] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "178453a42e9f433aafb200943b954a52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } } }
I20260620 15:17:08.175812 8488 tablet_replica.cc:442] T 39e6fe184218422fbdb7ab93595384d0 P a8a97c74af9844d28027d062ec3d7851: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:17:08.184044 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.176862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.0.97.65:37463)
0620 15:17:08.177689 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:08.177696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:08.177717 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:17:08.177812 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:08.177818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:08.177946 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:08.178102 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:17:08.178417 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.179567 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.181340 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.181958 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.182097 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.182528 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:17:08.183274 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:17:08.183419 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:08.183638 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 15:17:08.183717 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718}
I20260620 15:17:08.200269 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.194170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:35685 (local address 127.0.97.66:39651)
0620 15:17:08.194395 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:08.194401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:08.194932 (+ 531us) server_negotiation.cc:407] Connection header received
0620 15:17:08.195096 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:08.195102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:08.195179 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:08.195279 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:17:08.195735 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.196889 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.198242 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.198910 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.199114 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.199535 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:17:08.199697 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:17:08.199794 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:08.199893 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 15:17:08.199975 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:17:08.200394 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.194126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39651 (local address 127.0.97.65:35685)
0620 15:17:08.194815 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:08.194838 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:08.194973 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:08.195296 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:08.195299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:08.195308 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:17:08.195598 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:08.195605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.197024 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.197028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:08.198088 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:08.198099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.199157 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.199166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:08.199299 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.199351 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:17:08.199795 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:17:08.199803 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:17:08.199865 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 15:17:08.200043 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":401,"threads_started":1}
I20260620 15:17:08.209776 8408 raft_consensus.cc:1261] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Refusing update from remote peer a8a97c74af9844d28027d062ec3d7851: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:08.209947 8274 raft_consensus.cc:1261] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Refusing update from remote peer a8a97c74af9844d28027d062ec3d7851: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:08.210469 8513 consensus_queue.cc:1035] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:08.211067 8488 consensus_queue.cc:1035] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:08.254987 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.248162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39493 (local address 127.0.97.65:37463)
0620 15:17:08.248332 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:08.248338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:08.249103 (+ 765us) server_negotiation.cc:407] Connection header received
0620 15:17:08.249422 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:08.249426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:08.249492 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:08.249592 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:17:08.250272 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.251086 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.253145 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.253622 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.253746 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.254387 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:17:08.254502 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:17:08.254555 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:08.254734 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 15:17:08.254794 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:17:08.255185 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.248184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37463 (local address 127.0.97.66:39493)
0620 15:17:08.248992 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:08.249010 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:08.249173 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:08.249603 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:08.249609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:08.249622 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:17:08.250084 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:08.250096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.251205 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.251211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:08.252999 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:08.253012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.253980 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.253986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:08.254154 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.254214 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:17:08.254624 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:17:08.254640 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 15:17:08.254708 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 15:17:08.254846 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":406,"threads_started":1}
I20260620 15:17:08.294627 7972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:08.294852 7972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:08.298424 7972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da: Sending DeleteTablet for 3 replicas of tablet 9a71c61331ad415daaf31b4d1e62fbfb
I20260620 15:17:08.299357 8388 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a71c61331ad415daaf31b4d1e62fbfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:08 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:37170
I20260620 15:17:08.299427 8110 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a71c61331ad415daaf31b4d1e62fbfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:08 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:47500
I20260620 15:17:08.299631 8505 tablet_replica.cc:330] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae: stopping tablet replica
I20260620 15:17:08.299822 8505 raft_consensus.cc:2229] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:08.299885 8249 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a71c61331ad415daaf31b4d1e62fbfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:08 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:33518
I20260620 15:17:08.299998 8505 raft_consensus.cc:2258] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:08.300163 8507 tablet_replica.cc:330] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52: stopping tablet replica
I20260620 15:17:08.300215 8506 tablet_replica.cc:330] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851: stopping tablet replica
I20260620 15:17:08.300351 8507 raft_consensus.cc:2229] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:08.300381 8506 raft_consensus.cc:2229] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:17:08.300571 8507 raft_consensus.cc:2258] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:08.301028 8506 raft_consensus.cc:2258] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:08.302078 8507 ts_tablet_manager.cc:1905] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:08.302320 8505 ts_tablet_manager.cc:1905] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:08.305338 8506 ts_tablet_manager.cc:1905] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:08.309016 8505 ts_tablet_manager.cc:1918] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:17:08.309161 8505 log.cc:1196] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/9a71c61331ad415daaf31b4d1e62fbfb
I20260620 15:17:08.309562 8505 ts_tablet_manager.cc:1939] T 9a71c61331ad415daaf31b4d1e62fbfb P ee235e52a7074d2a877bfbffac6cceae: Deleting consensus metadata
I20260620 15:17:08.310842 7958 catalog_manager.cc:4841] TS ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921): tablet 9a71c61331ad415daaf31b4d1e62fbfb (table test-table [id=7b8ef2e20f3c4dc697c31963f23d0815]) successfully deleted
I20260620 15:17:08.318779 8506 ts_tablet_manager.cc:1918] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:17:08.318940 8506 log.cc:1196] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/9a71c61331ad415daaf31b4d1e62fbfb
I20260620 15:17:08.319402 8506 ts_tablet_manager.cc:1939] T 9a71c61331ad415daaf31b4d1e62fbfb P a8a97c74af9844d28027d062ec3d7851: Deleting consensus metadata
I20260620 15:17:08.320641 7959 catalog_manager.cc:4841] TS a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463): tablet 9a71c61331ad415daaf31b4d1e62fbfb (table test-table [id=7b8ef2e20f3c4dc697c31963f23d0815]) successfully deleted
I20260620 15:17:08.325452 8507 ts_tablet_manager.cc:1918] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:17:08.325600 8507 log.cc:1196] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/9a71c61331ad415daaf31b4d1e62fbfb
I20260620 15:17:08.325989 8507 ts_tablet_manager.cc:1939] T 9a71c61331ad415daaf31b4d1e62fbfb P 178453a42e9f433aafb200943b954a52: Deleting consensus metadata
I20260620 15:17:08.326951 7959 catalog_manager.cc:4841] TS 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651): tablet 9a71c61331ad415daaf31b4d1e62fbfb (table test-table [id=7b8ef2e20f3c4dc697c31963f23d0815]) successfully deleted
Jun 20 15:17:08 dist-test-slave-7s64 krb5kdc[7929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:17:08 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:08 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:17:08.366170 8022 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:08.356375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.0.97.126:38049)
0620 15:17:08.356649 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:08.356659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:08.357155 (+ 496us) server_negotiation.cc:407] Connection header received
0620 15:17:08.357351 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:08.357357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:08.357430 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:08.357532 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:08.358768 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.359559 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.360928 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.361315 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.361443 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.365884 (+ 4441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55792: BlockingRecv error: recv got EOF from 127.0.0.1:55792 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 15:17:08 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260620 15:17:08.383531 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.370011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.0.97.126:38049)
0620 15:17:08.370187 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:08.370192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:08.370874 (+ 682us) server_negotiation.cc:407] Connection header received
0620 15:17:08.371055 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:08.371078 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:08.371154 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:08.371256 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:08.372337 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.373100 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.374343 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.374737 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.374877 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.378434 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:08.378439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:08.378443 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:08.378481 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:08.381071 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:08.381672 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:08.381679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:08.381682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:08.381804 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:08.382177 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:08.382184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:08.382187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:08.382448 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:08.382691 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:08.382973 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 15:17:08.383175 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:17:08.387077 7972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:17:08.387639 7972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:17:08.394230 8110 tablet_service.cc:1467] Processing CreateTablet for tablet 2df36e5048b9487da2adf635210619c6 (DEFAULT_TABLE table=test-table [id=57f40dda8a9e4ebea0eaa9e826ae44b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:08.394536 8388 tablet_service.cc:1467] Processing CreateTablet for tablet 2df36e5048b9487da2adf635210619c6 (DEFAULT_TABLE table=test-table [id=57f40dda8a9e4ebea0eaa9e826ae44b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:08.394667 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df36e5048b9487da2adf635210619c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:08.394901 8388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df36e5048b9487da2adf635210619c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:08.401273 8249 tablet_service.cc:1467] Processing CreateTablet for tablet 2df36e5048b9487da2adf635210619c6 (DEFAULT_TABLE table=test-table [id=57f40dda8a9e4ebea0eaa9e826ae44b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:08.401500 8484 tablet_bootstrap.cc:492] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851: Bootstrap starting.
I20260620 15:17:08.401716 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2df36e5048b9487da2adf635210619c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:08.403144 8484 tablet_bootstrap.cc:654] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:08.407248 8483 tablet_bootstrap.cc:492] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae: Bootstrap starting.
I20260620 15:17:08.408780 8483 tablet_bootstrap.cc:654] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:08.409446 8482 tablet_bootstrap.cc:492] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52: Bootstrap starting.
I20260620 15:17:08.410992 8482 tablet_bootstrap.cc:654] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:08.421494 8482 tablet_bootstrap.cc:492] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52: No bootstrap required, opened a new log
I20260620 15:17:08.421685 8482 ts_tablet_manager.cc:1397] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260620 15:17:08.422470 8482 raft_consensus.cc:348] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.422722 8483 tablet_bootstrap.cc:492] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae: No bootstrap required, opened a new log
I20260620 15:17:08.422865 8483 ts_tablet_manager.cc:1397] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20260620 15:17:08.422865 8482 raft_consensus.cc:374] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:08.423075 8482 raft_consensus.cc:729] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178453a42e9f433aafb200943b954a52, State: Initialized, Role: FOLLOWER
I20260620 15:17:08.423457 8482 consensus_queue.cc:260] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.423595 8483 raft_consensus.cc:348] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.423852 8483 raft_consensus.cc:374] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:08.423934 8483 raft_consensus.cc:729] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee235e52a7074d2a877bfbffac6cceae, State: Initialized, Role: FOLLOWER
I20260620 15:17:08.424175 8483 consensus_queue.cc:260] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.424749 8483 ts_tablet_manager.cc:1428] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 15:17:08.426994 8482 ts_tablet_manager.cc:1428] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260620 15:17:08.428171 8484 tablet_bootstrap.cc:492] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851: No bootstrap required, opened a new log
I20260620 15:17:08.428340 8484 ts_tablet_manager.cc:1397] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.004s
I20260620 15:17:08.429116 8484 raft_consensus.cc:348] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.429401 8484 raft_consensus.cc:374] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:08.429518 8484 raft_consensus.cc:729] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a97c74af9844d28027d062ec3d7851, State: Initialized, Role: FOLLOWER
I20260620 15:17:08.429754 8484 consensus_queue.cc:260] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.430294 8484 ts_tablet_manager.cc:1428] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 15:17:08.489154 8490 raft_consensus.cc:482] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:17:08.489377 8490 raft_consensus.cc:504] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.491066 8490 leader_election.cc:288] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463), 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651)
I20260620 15:17:08.500958 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.491622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:47839 (local address 127.0.97.65:37463)
0620 15:17:08.491809 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:08.491816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:08.492264 (+ 448us) server_negotiation.cc:407] Connection header received
0620 15:17:08.492562 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:08.492568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:08.492643 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:08.492705 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:08.493339 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.494455 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.498297 (+ 3842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.499733 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.499848 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.499947 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:08.500525 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 15:17:08.500660 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:17:08.501848 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.491501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37463 (local address 127.0.97.67:47839)
0620 15:17:08.492165 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:08.492184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:08.492376 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:08.492893 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:08.492897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:08.492925 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:08.493178 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:08.493187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.494632 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.494640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:08.498115 (+ 3475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:08.498130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.500130 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.500136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:08.500320 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.500355 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:17:08.501383 (+ 1028us) client_negotiation.cc:240] Negotiation successful
0620 15:17:08.501520 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":408,"threads_started":1}
I20260620 15:17:08.503019 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df36e5048b9487da2adf635210619c6" candidate_uuid: "ee235e52a7074d2a877bfbffac6cceae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a97c74af9844d28027d062ec3d7851" is_pre_election: true
I20260620 15:17:08.503270 8130 raft_consensus.cc:2454] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee235e52a7074d2a877bfbffac6cceae in term 0.
I20260620 15:17:08.503758 8340 leader_election.cc:302] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8a97c74af9844d28027d062ec3d7851, ee235e52a7074d2a877bfbffac6cceae; no voters:
I20260620 15:17:08.504134 8490 raft_consensus.cc:2781] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:17:08.504236 8490 raft_consensus.cc:482] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:17:08.504316 8490 raft_consensus.cc:3037] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:08.507550 8490 raft_consensus.cc:504] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.508172 8490 leader_election.cc:288] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [CANDIDATE]: Term 1 election: Requested vote from peers a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463), 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651)
I20260620 15:17:08.508761 8130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df36e5048b9487da2adf635210619c6" candidate_uuid: "ee235e52a7074d2a877bfbffac6cceae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a97c74af9844d28027d062ec3d7851"
I20260620 15:17:08.508965 8130 raft_consensus.cc:3037] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:08.509297 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.492651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:45063 (local address 127.0.97.66:39651)
0620 15:17:08.492813 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:08.492819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:08.502298 (+ 9479us) server_negotiation.cc:407] Connection header received
0620 15:17:08.502531 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:08.502537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:08.502609 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:08.502674 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:08.503176 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.504296 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.506791 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.508100 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.508193 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.508303 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:08.508884 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 15:17:08.509019 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:17:08.510006 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.492259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39651 (local address 127.0.97.67:45063)
0620 15:17:08.502211 (+ 9952us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:08.502230 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:08.502391 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:08.502851 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:08.502856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:08.502879 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:08.503017 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:08.503026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.504631 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.504636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:08.506614 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:08.506628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.508520 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.508526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:08.508709 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.508741 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:17:08.509589 (+ 848us) client_negotiation.cc:240] Negotiation successful
0620 15:17:08.509725 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9857,"thread_start_us":10149,"threads_started":1}
I20260620 15:17:08.511962 8130 raft_consensus.cc:2454] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee235e52a7074d2a877bfbffac6cceae in term 1.
I20260620 15:17:08.512574 8340 leader_election.cc:302] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8a97c74af9844d28027d062ec3d7851, ee235e52a7074d2a877bfbffac6cceae; no voters:
I20260620 15:17:08.512813 8490 raft_consensus.cc:2781] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:08.515527 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df36e5048b9487da2adf635210619c6" candidate_uuid: "ee235e52a7074d2a877bfbffac6cceae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178453a42e9f433aafb200943b954a52" is_pre_election: true
I20260620 15:17:08.515741 8274 raft_consensus.cc:2454] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee235e52a7074d2a877bfbffac6cceae in term 0.
I20260620 15:17:08.516424 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2df36e5048b9487da2adf635210619c6" candidate_uuid: "ee235e52a7074d2a877bfbffac6cceae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178453a42e9f433aafb200943b954a52"
I20260620 15:17:08.516621 8274 raft_consensus.cc:3037] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:08.519378 8274 raft_consensus.cc:2454] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee235e52a7074d2a877bfbffac6cceae in term 1.
I20260620 15:17:08.526502 8490 raft_consensus.cc:686] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 1 LEADER]: Becoming Leader. State: Replica: ee235e52a7074d2a877bfbffac6cceae, State: Running, Role: LEADER
I20260620 15:17:08.526878 8490 consensus_queue.cc:237] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.542299 7972 catalog_manager.cc:5495] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae reported cstate change: term changed from 0 to 1, leader changed from <none> to ee235e52a7074d2a877bfbffac6cceae (127.0.97.67). New cstate: current_term: 1 leader_uuid: "ee235e52a7074d2a877bfbffac6cceae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } health_report { overall_health: HEALTHY } } }
I20260620 15:17:08.583968 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.575206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.0.97.67:32921)
0620 15:17:08.576907 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:08.576912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:08.576945 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:17:08.577035 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:08.577041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:08.577140 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:08.577298 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:17:08.577730 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.578933 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.580993 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.581636 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.581757 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.582301 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:17:08.582978 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:17:08.583505 (+ 527us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:08.583553 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 15:17:08.583628 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1572,"thread_start_us":1446,"threads_started":1}
I20260620 15:17:08.587782 8274 raft_consensus.cc:1261] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Refusing update from remote peer ee235e52a7074d2a877bfbffac6cceae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:08.587983 8130 raft_consensus.cc:1261] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Refusing update from remote peer ee235e52a7074d2a877bfbffac6cceae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:08.588470 8557 consensus_queue.cc:1035] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [LEADER]: Connected to new peer: Peer: permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:08.589439 8557 consensus_queue.cc:1035] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:08.621592 7972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:08.621855 7972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:08.625208 7972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da: Sending DeleteTablet for 3 replicas of tablet 2df36e5048b9487da2adf635210619c6
I20260620 15:17:08.625944 8110 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df36e5048b9487da2adf635210619c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:08 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:47500
I20260620 15:17:08.626269 8506 tablet_replica.cc:330] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851: stopping tablet replica
I20260620 15:17:08.626483 8506 raft_consensus.cc:2229] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:08.626736 8506 raft_consensus.cc:2258] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:08.626926 8249 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df36e5048b9487da2adf635210619c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:08 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:33518
I20260620 15:17:08.627189 8507 tablet_replica.cc:330] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52: stopping tablet replica
I20260620 15:17:08.627233 8388 tablet_service.cc:1514] Processing DeleteTablet for tablet 2df36e5048b9487da2adf635210619c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:08 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:37170
I20260620 15:17:08.627354 8507 raft_consensus.cc:2229] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:08.627487 8505 tablet_replica.cc:330] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae: stopping tablet replica
I20260620 15:17:08.627599 8507 raft_consensus.cc:2258] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:08.627658 8505 raft_consensus.cc:2229] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:17:08.628104 8505 raft_consensus.cc:2258] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:08.628364 7973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 15:17:08.628582 8506 ts_tablet_manager.cc:1905] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 15:17:08.629005 7973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:17:08.629393 8507 ts_tablet_manager.cc:1905] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:08.632614 8505 ts_tablet_manager.cc:1905] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:08.634759 8506 ts_tablet_manager.cc:1918] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:08.634888 8506 log.cc:1196] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/2df36e5048b9487da2adf635210619c6
I20260620 15:17:08.635293 8506 ts_tablet_manager.cc:1939] T 2df36e5048b9487da2adf635210619c6 P a8a97c74af9844d28027d062ec3d7851: Deleting consensus metadata
I20260620 15:17:08.636569 8388 tablet_service.cc:1467] Processing CreateTablet for tablet 40674a6193fe434fa04489a46442fa5c (DEFAULT_TABLE table=test-table [id=acc60c3f926f4c0eabc36d57d2ba5220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:08.636605 8110 tablet_service.cc:1467] Processing CreateTablet for tablet 40674a6193fe434fa04489a46442fa5c (DEFAULT_TABLE table=test-table [id=acc60c3f926f4c0eabc36d57d2ba5220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:08.636974 8388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40674a6193fe434fa04489a46442fa5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:08.636984 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40674a6193fe434fa04489a46442fa5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:08.639809 8249 tablet_service.cc:1467] Processing CreateTablet for tablet 40674a6193fe434fa04489a46442fa5c (DEFAULT_TABLE table=test-table [id=acc60c3f926f4c0eabc36d57d2ba5220]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:08.640285 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40674a6193fe434fa04489a46442fa5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:08.640739 7959 catalog_manager.cc:4841] TS a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463): tablet 2df36e5048b9487da2adf635210619c6 (table test-table [id=57f40dda8a9e4ebea0eaa9e826ae44b1]) successfully deleted
I20260620 15:17:08.645028 8484 tablet_bootstrap.cc:492] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851: Bootstrap starting.
I20260620 15:17:08.646796 8484 tablet_bootstrap.cc:654] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:08.658332 8505 ts_tablet_manager.cc:1918] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:08.660516 8483 tablet_bootstrap.cc:492] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae: Bootstrap starting.
I20260620 15:17:08.661885 8484 tablet_bootstrap.cc:492] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851: No bootstrap required, opened a new log
I20260620 15:17:08.662066 8484 ts_tablet_manager.cc:1397] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.007s
I20260620 15:17:08.662483 8483 tablet_bootstrap.cc:654] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:08.662868 8484 raft_consensus.cc:348] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.665407 8505 log.cc:1196] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/2df36e5048b9487da2adf635210619c6
I20260620 15:17:08.665876 8505 ts_tablet_manager.cc:1939] T 2df36e5048b9487da2adf635210619c6 P ee235e52a7074d2a877bfbffac6cceae: Deleting consensus metadata
I20260620 15:17:08.666894 7958 catalog_manager.cc:4841] TS ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921): tablet 2df36e5048b9487da2adf635210619c6 (table test-table [id=57f40dda8a9e4ebea0eaa9e826ae44b1]) successfully deleted
I20260620 15:17:08.667136 8507 ts_tablet_manager.cc:1918] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:08.667279 8507 log.cc:1196] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/2df36e5048b9487da2adf635210619c6
I20260620 15:17:08.667672 8507 ts_tablet_manager.cc:1939] T 2df36e5048b9487da2adf635210619c6 P 178453a42e9f433aafb200943b954a52: Deleting consensus metadata
I20260620 15:17:08.668629 7959 catalog_manager.cc:4841] TS 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651): tablet 2df36e5048b9487da2adf635210619c6 (table test-table [id=57f40dda8a9e4ebea0eaa9e826ae44b1]) successfully deleted
I20260620 15:17:08.668641 8482 tablet_bootstrap.cc:492] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52: Bootstrap starting.
I20260620 15:17:08.670351 8482 tablet_bootstrap.cc:654] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:08.672575 8484 raft_consensus.cc:374] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:08.672735 8484 raft_consensus.cc:729] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a97c74af9844d28027d062ec3d7851, State: Initialized, Role: FOLLOWER
I20260620 15:17:08.673035 8484 consensus_queue.cc:260] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.673661 8484 ts_tablet_manager.cc:1428] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20260620 15:17:08.685724 8483 tablet_bootstrap.cc:492] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae: No bootstrap required, opened a new log
I20260620 15:17:08.685889 8483 ts_tablet_manager.cc:1397] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.007s
I20260620 15:17:08.687536 8483 raft_consensus.cc:348] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.687824 8483 raft_consensus.cc:374] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:08.687909 8483 raft_consensus.cc:729] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee235e52a7074d2a877bfbffac6cceae, State: Initialized, Role: FOLLOWER
I20260620 15:17:08.689481 8483 consensus_queue.cc:260] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.690338 8482 tablet_bootstrap.cc:492] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52: No bootstrap required, opened a new log
I20260620 15:17:08.690526 8482 ts_tablet_manager.cc:1397] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.004s
I20260620 15:17:08.691306 8482 raft_consensus.cc:348] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.691573 8482 raft_consensus.cc:374] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:08.691660 8482 raft_consensus.cc:729] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178453a42e9f433aafb200943b954a52, State: Initialized, Role: FOLLOWER
I20260620 15:17:08.691924 8482 consensus_queue.cc:260] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.695212 8482 ts_tablet_manager.cc:1428] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260620 15:17:08.700049 8483 ts_tablet_manager.cc:1428] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20260620 15:17:08.881812 8534 raft_consensus.cc:482] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:17:08.882134 8534 raft_consensus.cc:504] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.883159 8534 leader_election.cc:288] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651), ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921)
I20260620 15:17:08.884070 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40674a6193fe434fa04489a46442fa5c" candidate_uuid: "a8a97c74af9844d28027d062ec3d7851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178453a42e9f433aafb200943b954a52" is_pre_election: true
I20260620 15:17:08.884245 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40674a6193fe434fa04489a46442fa5c" candidate_uuid: "a8a97c74af9844d28027d062ec3d7851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee235e52a7074d2a877bfbffac6cceae" is_pre_election: true
I20260620 15:17:08.884383 8274 raft_consensus.cc:2454] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a97c74af9844d28027d062ec3d7851 in term 0.
I20260620 15:17:08.884622 8408 raft_consensus.cc:2454] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a97c74af9844d28027d062ec3d7851 in term 0.
I20260620 15:17:08.885017 8062 leader_election.cc:302] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178453a42e9f433aafb200943b954a52, a8a97c74af9844d28027d062ec3d7851; no voters:
I20260620 15:17:08.885527 8534 raft_consensus.cc:2781] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:17:08.885703 8534 raft_consensus.cc:482] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:17:08.885844 8534 raft_consensus.cc:3037] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:08.888413 8534 raft_consensus.cc:504] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.889643 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40674a6193fe434fa04489a46442fa5c" candidate_uuid: "a8a97c74af9844d28027d062ec3d7851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "178453a42e9f433aafb200943b954a52"
I20260620 15:17:08.889739 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40674a6193fe434fa04489a46442fa5c" candidate_uuid: "a8a97c74af9844d28027d062ec3d7851" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee235e52a7074d2a877bfbffac6cceae"
I20260620 15:17:08.889876 8274 raft_consensus.cc:3037] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:08.889971 8408 raft_consensus.cc:3037] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:08.893340 8274 raft_consensus.cc:2454] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a97c74af9844d28027d062ec3d7851 in term 1.
I20260620 15:17:08.893863 8408 raft_consensus.cc:2454] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a97c74af9844d28027d062ec3d7851 in term 1.
I20260620 15:17:08.893934 8062 leader_election.cc:302] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 178453a42e9f433aafb200943b954a52, a8a97c74af9844d28027d062ec3d7851; no voters:
I20260620 15:17:08.894505 8534 leader_election.cc:288] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [CANDIDATE]: Term 1 election: Requested vote from peers 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651), ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921)
I20260620 15:17:08.894760 8534 raft_consensus.cc:2781] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:08.895084 8534 raft_consensus.cc:686] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 1 LEADER]: Becoming Leader. State: Replica: a8a97c74af9844d28027d062ec3d7851, State: Running, Role: LEADER
I20260620 15:17:08.895607 8534 consensus_queue.cc:237] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } }
I20260620 15:17:08.902544 7972 catalog_manager.cc:5495] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8a97c74af9844d28027d062ec3d7851 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "a8a97c74af9844d28027d062ec3d7851" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a97c74af9844d28027d062ec3d7851" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 } health_report { overall_health: UNKNOWN } } }
I20260620 15:17:08.958082 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:08.952834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.0.97.65:37463)
0620 15:17:08.953033 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:08.953039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:08.953178 (+ 139us) server_negotiation.cc:407] Connection header received
0620 15:17:08.953310 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:08.953315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:08.953394 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:08.953485 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:17:08.953819 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.954943 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.956207 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:08.956856 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:08.957003 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:08.957397 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:17:08.957532 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:17:08.957626 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:08.957717 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 15:17:08.957793 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:17:08.967005 8274 raft_consensus.cc:1261] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Refusing update from remote peer a8a97c74af9844d28027d062ec3d7851: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:08.967598 8567 consensus_queue.cc:1035] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "178453a42e9f433aafb200943b954a52" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:08.968163 8408 raft_consensus.cc:1261] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Refusing update from remote peer a8a97c74af9844d28027d062ec3d7851: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:08.968732 8534 consensus_queue.cc:1035] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee235e52a7074d2a877bfbffac6cceae" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 32921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:09.054992 7972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:09.055238 7972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:09.057916 7972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f15bd72a16204aa4818d02a2e96245da: Sending DeleteTablet for 3 replicas of tablet 40674a6193fe434fa04489a46442fa5c
I20260620 15:17:09.058923 8388 tablet_service.cc:1514] Processing DeleteTablet for tablet 40674a6193fe434fa04489a46442fa5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:09 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:37170
I20260620 15:17:09.058883 8249 tablet_service.cc:1514] Processing DeleteTablet for tablet 40674a6193fe434fa04489a46442fa5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:09 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:33518
I20260620 15:17:09.059192 8505 tablet_replica.cc:330] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae: stopping tablet replica
I20260620 15:17:09.059206 8110 tablet_service.cc:1514] Processing DeleteTablet for tablet 40674a6193fe434fa04489a46442fa5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:09 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:47500
I20260620 15:17:09.059382 8505 raft_consensus.cc:2229] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:09.059504 8506 tablet_replica.cc:330] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851: stopping tablet replica
I20260620 15:17:09.059583 8505 raft_consensus.cc:2258] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:09.059720 8506 raft_consensus.cc:2229] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:17:09.059864 8507 tablet_replica.cc:330] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52: stopping tablet replica
I20260620 15:17:09.060016 8506 raft_consensus.cc:2258] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:09.060041 8507 raft_consensus.cc:2229] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:09.060184 8507 raft_consensus.cc:2258] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:09.060895 8505 ts_tablet_manager.cc:1905] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:09.061134 8506 ts_tablet_manager.cc:1905] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:09.065335 8507 ts_tablet_manager.cc:1905] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:09.067831 8505 ts_tablet_manager.cc:1918] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:17:09.067971 8505 log.cc:1196] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/40674a6193fe434fa04489a46442fa5c
I20260620 15:17:09.068375 8505 ts_tablet_manager.cc:1939] T 40674a6193fe434fa04489a46442fa5c P ee235e52a7074d2a877bfbffac6cceae: Deleting consensus metadata
I20260620 15:17:09.069643 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 8045
I20260620 15:17:09.070700 7958 catalog_manager.cc:4841] TS ee235e52a7074d2a877bfbffac6cceae (127.0.97.67:32921): tablet 40674a6193fe434fa04489a46442fa5c (table test-table [id=acc60c3f926f4c0eabc36d57d2ba5220]) successfully deleted
I20260620 15:17:09.073849 8507 ts_tablet_manager.cc:1918] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:17:09.073992 8507 log.cc:1196] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/40674a6193fe434fa04489a46442fa5c
I20260620 15:17:09.074368 8507 ts_tablet_manager.cc:1939] T 40674a6193fe434fa04489a46442fa5c P 178453a42e9f433aafb200943b954a52: Deleting consensus metadata
I20260620 15:17:09.075258 7959 catalog_manager.cc:4841] TS 178453a42e9f433aafb200943b954a52 (127.0.97.66:39651): tablet 40674a6193fe434fa04489a46442fa5c (table test-table [id=acc60c3f926f4c0eabc36d57d2ba5220]) successfully deleted
I20260620 15:17:09.077518 8506 ts_tablet_manager.cc:1918] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:17:09.077699 8506 log.cc:1196] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/40674a6193fe434fa04489a46442fa5c
I20260620 15:17:09.078100 8506 ts_tablet_manager.cc:1939] T 40674a6193fe434fa04489a46442fa5c P a8a97c74af9844d28027d062ec3d7851: Deleting consensus metadata
I20260620 15:17:09.079113 7959 catalog_manager.cc:4841] TS a8a97c74af9844d28027d062ec3d7851 (127.0.97.65:37463): tablet 40674a6193fe434fa04489a46442fa5c (table test-table [id=acc60c3f926f4c0eabc36d57d2ba5220]) successfully deleted
Jun 20 15:17:09 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.65@KRBTEST.COM
I20260620 15:17:09.087508 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:09.070770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.0.97.65:37463)
0620 15:17:09.071085 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:09.071091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:09.071378 (+ 287us) server_negotiation.cc:407] Connection header received
0620 15:17:09.071638 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:09.071645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:09.071749 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:09.071856 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:09.073401 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:09.074520 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:09.076348 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:09.076973 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:09.077093 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:09.081615 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:09.081627 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:09.081641 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:09.081668 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:09.084717 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:09.085378 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:09.085389 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:09.085393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:09.085610 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:09.085974 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:09.085980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:09.085984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:09.086322 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:09.086655 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:09.086940 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 15:17:09.087142 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260620 15:17:09.088120 8175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:09.246659 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 8045
I20260620 15:17:09.268988 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 8184
Jun 20 15:17:09 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.66@KRBTEST.COM
I20260620 15:17:09.284164 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:09.269706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.0.97.66:39651)
0620 15:17:09.270182 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:09.270189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:09.270213 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:17:09.270262 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:09.270265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:09.270326 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:09.270440 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:09.271623 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:09.272368 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:09.273665 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:09.274091 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:09.274224 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:09.278246 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:09.278267 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:09.278282 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:09.278316 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:09.281243 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:09.281930 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:09.281937 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:09.281939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:09.282099 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:09.282497 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:09.282504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:09.282508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:09.282869 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:09.283187 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:09.283522 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 15:17:09.283748 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":223,"threads_started":1}
I20260620 15:17:09.284970 8314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:09.424831 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 8184
I20260620 15:17:09.449302 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 8323
Jun 20 15:17:09 dist-test-slave-7s64 krb5kdc[7929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.67@KRBTEST.COM
I20260620 15:17:09.464289 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:09.450027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.0.97.67:32921)
0620 15:17:09.450472 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:09.450476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:09.450512 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:17:09.450580 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:09.450583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:09.450639 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:09.450731 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:09.451919 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:09.452658 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:09.453916 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:09.454333 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:09.454423 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:09.458921 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:09.458935 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:09.458955 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:09.458992 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:09.461782 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:09.462361 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:09.462369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:09.462371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:09.462532 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:09.462845 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:09.462848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:09.462850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:09.463199 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:09.463467 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:09.463691 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 15:17:09.463887 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":203,"threads_started":1}
I20260620 15:17:09.464954 8453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:09.588943 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 8323
I20260620 15:17:09.607754 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 7942
I20260620 15:17:09.608930 8014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:09.720388 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 7942
2026-06-20T15:17:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5448 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:17:09 dist-test-slave-7s64 krb5kdc[8586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:17:09 dist-test-slave-7s64 krb5kdc[8586](info): set up 2 sockets
Jun 20 15:17:09 dist-test-slave-7s64 krb5kdc[8586](info): commencing operation
krb5kdc: starting...
W20260620 15:17:10.829535 389 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:17:10 dist-test-slave-7s64 krb5kdc[8586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:17:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:17:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:17:11.022218 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39929
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:39929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:11.330416 8599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:11.330668 8599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:11.330835 8599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:11.337836 8599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:17:11.337924 8599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:11.338002 8599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:11.338104 8599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:17:11.338168 8599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:17:11.347328 8599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:39929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:11.348466 8599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:11.349674 8599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:11.360670 8607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:11.360745 8604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:11.360720 8605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:11.361582 8599 server_base.cc:1034] running on GCE node
I20260620 15:17:11.362282 8599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:11.363587 8599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:11.364813 8599 hybrid_clock.cc:648] HybridClock initialized: now 1781968631364759 us; error 86 us; skew 500 ppm
I20260620 15:17:11.365142 8599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T15:17:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1714 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:17:11 dist-test-slave-7s64 krb5kdc[8613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:17:11 dist-test-slave-7s64 krb5kdc[8613](info): set up 2 sockets
Jun 20 15:17:11 dist-test-slave-7s64 krb5kdc[8613](info): commencing operation
krb5kdc: starting...
W20260620 15:17:12.517448 389 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:17:12 dist-test-slave-7s64 krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:17:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:17:12.713384 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35025
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:13.016422 8626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:13.016705 8626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:13.016916 8626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:13.023789 8626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:17:13.023875 8626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:13.023984 8626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:13.024082 8626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:17:13.024168 8626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:17:13.033203 8626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:13.034369 8626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:13.035585 8626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:13.046809 8632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:13.046904 8631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:13.047348 8634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:13.047462 8626 server_base.cc:1034] running on GCE node
I20260620 15:17:13.048188 8626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:13.049427 8626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:13.050622 8626 hybrid_clock.cc:648] HybridClock initialized: now 1781968633050590 us; error 29 us; skew 500 ppm
I20260620 15:17:13.050937 8626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:13 dist-test-slave-7s64 krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:13.057986 8626 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:17:13.059502 8626 webserver.cc:460] Webserver started at http://127.0.97.126:33391/ using document root <none> and password file <none>
I20260620 15:17:13.060220 8626 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:13.060454 8626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:13.060853 8626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:13.063902 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "b55563cf44124ffd8e43badb764b88bd"
format_stamp: "Formatted at 2026-06-20 15:17:13 on dist-test-slave-7s64"
server_key: "06332b73ea6816b79bd5c43fc2a40dcf"
server_key_iv: "17ef7e37c469141a70dfdf4cd8faa068"
server_key_version: "encryptionkey@0"
I20260620 15:17:13.064800 8626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "b55563cf44124ffd8e43badb764b88bd"
format_stamp: "Formatted at 2026-06-20 15:17:13 on dist-test-slave-7s64"
server_key: "06332b73ea6816b79bd5c43fc2a40dcf"
server_key_iv: "17ef7e37c469141a70dfdf4cd8faa068"
server_key_version: "encryptionkey@0"
I20260620 15:17:13.069861 8626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 15:17:13.073347 8641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:13.074563 8626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:17:13.074744 8626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "b55563cf44124ffd8e43badb764b88bd"
format_stamp: "Formatted at 2026-06-20 15:17:13 on dist-test-slave-7s64"
server_key: "06332b73ea6816b79bd5c43fc2a40dcf"
server_key_iv: "17ef7e37c469141a70dfdf4cd8faa068"
server_key_version: "encryptionkey@0"
I20260620 15:17:13.074955 8626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:13.106200 8626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:13.110718 8626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:13.111006 8626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:13.129827 8626 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35025
I20260620 15:17:13.129854 8692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35025 every 8 connection(s)
I20260620 15:17:13.131150 8626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:17:13.135576 8693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:13.139847 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 8626
I20260620 15:17:13.140084 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:17:13.140964 389 external_mini_cluster.cc:1428] Setting key 2c190159c0423c9db1ffee15e88e27e5
I20260620 15:17:13.148979 8693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd: Bootstrap starting.
I20260620 15:17:13.151767 8693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:13.152745 8693 log.cc:824] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:17:13 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:17:13.155267 8693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd: No bootstrap required, opened a new log
I20260620 15:17:13.162029 8696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:13.143442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.0.97.126:35025)
0620 15:17:13.144182 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:13.144195 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:13.144232 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:17:13.145801 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:13.145821 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:13.146189 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:13.146559 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:13.147974 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:13.149323 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:13.150590 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:13.151217 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:13.151403 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:13.155179 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:13.155186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:13.155199 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:13.155230 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:13.158317 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:13.158928 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:13.158942 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:13.158948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:13.159101 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:13.159479 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:13.159489 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:13.159493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:13.159982 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:13.160290 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:13.160711 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 15:17:13.161093 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":331,"threads_started":1}
I20260620 15:17:13.163424 8693 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } }
I20260620 15:17:13.163760 8693 raft_consensus.cc:374] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:13.163877 8693 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55563cf44124ffd8e43badb764b88bd, State: Initialized, Role: FOLLOWER
I20260620 15:17:13.164443 8693 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } }
I20260620 15:17:13.164806 8693 raft_consensus.cc:388] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:17:13.164950 8693 raft_consensus.cc:482] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:17:13.165161 8693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:13.167084 8693 raft_consensus.cc:504] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } }
I20260620 15:17:13.167529 8693 leader_election.cc:302] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b55563cf44124ffd8e43badb764b88bd; no voters:
I20260620 15:17:13.168037 8693 leader_election.cc:288] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:17:13.168200 8698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:13.169390 8698 raft_consensus.cc:686] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 1 LEADER]: Becoming Leader. State: Replica: b55563cf44124ffd8e43badb764b88bd, State: Running, Role: LEADER
I20260620 15:17:13.169862 8698 consensus_queue.cc:237] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } }
I20260620 15:17:13.170008 8693 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:17:13.172475 8700 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55563cf44124ffd8e43badb764b88bd. Latest consensus state: current_term: 1 leader_uuid: "b55563cf44124ffd8e43badb764b88bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } } }
I20260620 15:17:13.173004 8700 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:13.172694 8699 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b55563cf44124ffd8e43badb764b88bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } } }
I20260620 15:17:13.173427 8699 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:13.174685 8705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:17:13.179579 8705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:17:13.187706 8705 catalog_manager.cc:1347] Generated new cluster ID: 0c5fca7b919648f2bc8fc63d02ce7476
I20260620 15:17:13.187805 8705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:17:13.223984 8705 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:17:13.225284 8705 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:17:13.234158 8705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd: Generated new TSK 0
I20260620 15:17:13.234869 8705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:17:13.293242 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35025
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:13.613729 8719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:13.614001 8719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:13.614145 8719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:13.622368 8719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:13.622469 8719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:13.622573 8719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:17:13.632014 8719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:13.633158 8719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:13.634377 8719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:13.647658 8724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:13.647603 8725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:13.647691 8727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:13.648456 8719 server_base.cc:1034] running on GCE node
I20260620 15:17:13.649068 8719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:13.649931 8719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:13.651125 8719 hybrid_clock.cc:648] HybridClock initialized: now 1781968633651091 us; error 35 us; skew 500 ppm
I20260620 15:17:13.651378 8719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:13 dist-test-slave-7s64 krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:13.654377 8719 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:17:13.655620 8719 webserver.cc:460] Webserver started at http://127.0.97.65:41073/ using document root <none> and password file <none>
I20260620 15:17:13.656209 8719 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:13.656301 8719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:13.656611 8719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:13.659210 8719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "0ddf68add0364ed9a84b596b0a425102"
format_stamp: "Formatted at 2026-06-20 15:17:13 on dist-test-slave-7s64"
server_key: "0ae19eb8637ebab92252fb782ef80e61"
server_key_iv: "ef2fe25c8dc1def9ff22e0e133e8acd7"
server_key_version: "encryptionkey@0"
I20260620 15:17:13.659897 8719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "0ddf68add0364ed9a84b596b0a425102"
format_stamp: "Formatted at 2026-06-20 15:17:13 on dist-test-slave-7s64"
server_key: "0ae19eb8637ebab92252fb782ef80e61"
server_key_iv: "ef2fe25c8dc1def9ff22e0e133e8acd7"
server_key_version: "encryptionkey@0"
I20260620 15:17:13.664917 8719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:17:13.668372 8734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:13.669793 8719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:17:13.669947 8719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "0ddf68add0364ed9a84b596b0a425102"
format_stamp: "Formatted at 2026-06-20 15:17:13 on dist-test-slave-7s64"
server_key: "0ae19eb8637ebab92252fb782ef80e61"
server_key_iv: "ef2fe25c8dc1def9ff22e0e133e8acd7"
server_key_version: "encryptionkey@0"
I20260620 15:17:13.670125 8719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:13.685575 8719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:13.689162 8719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:13.689373 8719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:13.690196 8719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:13.691345 8719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:13.691421 8719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:13.691498 8719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:13.691540 8719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:13.722463 8719 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:39537
I20260620 15:17:13.722545 8846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:39537 every 8 connection(s)
I20260620 15:17:13.723645 8719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:17:13.733803 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 8719
I20260620 15:17:13.734017 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:17:13.734638 389 external_mini_cluster.cc:1428] Setting key 20cbb492495490930878d15204d2244b
Jun 20 15:17:13 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:17:13.744208 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:13.726211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57267 (local address 127.0.97.126:35025)
0620 15:17:13.726659 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:13.726666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:13.727626 (+ 960us) server_negotiation.cc:407] Connection header received
0620 15:17:13.729452 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:13.729456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:13.729517 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:13.729611 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:13.730985 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:13.731684 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:13.733319 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:13.733771 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:13.733973 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:13.738227 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:13.738231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:13.738235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:13.738262 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:13.740750 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:13.741639 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:13.741643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:13.741645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:13.741762 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:13.742251 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:13.742255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:13.742257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:13.742497 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:13.742601 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:13.743395 (+ 794us) server_negotiation.cc:299] Negotiation successful
0620 15:17:13.743752 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":218,"threads_started":1}
I20260620 15:17:13.744763 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:13.726758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:57267)
0620 15:17:13.727465 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:13.727512 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:13.729208 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:13.729810 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:13.729819 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:13.730298 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:13.730841 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:13.730859 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:13.731820 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:13.731824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:13.733181 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:13.733194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:13.734091 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:13.734101 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:13.734448 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:13.735114 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:13.735166 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:13.738012 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:13.740895 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:13.740905 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:13.740923 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:13.741527 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:13.741869 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:13.741877 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:13.741884 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:13.742122 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:13.742619 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:13.742629 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:13.743031 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 15:17:13.743317 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 15:17:13.743728 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":268,"threads_started":1}
I20260620 15:17:13.746343 8847 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35025
I20260620 15:17:13.746641 8847 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:13.747291 8847 heartbeater.cc:511] Master 127.0.97.126:35025 requested a full tablet report, sending...
I20260620 15:17:13.749488 8658 ts_manager.cc:194] Registered new tserver with Master: 0ddf68add0364ed9a84b596b0a425102 (127.0.97.65:39537)
I20260620 15:17:13.751542 8658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:57267
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:17:13.781797 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35025
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:14.090327 8853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:14.090577 8853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:14.090727 8853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:14.097781 8853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:14.097868 8853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:14.098069 8853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:17:14.107069 8853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:14.108198 8853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:14.109408 8853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:14.122359 8858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:14.122325 8861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:14.122395 8859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:14.122942 8853 server_base.cc:1034] running on GCE node
I20260620 15:17:14.123546 8853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:14.124363 8853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:14.125551 8853 hybrid_clock.cc:648] HybridClock initialized: now 1781968634125518 us; error 46 us; skew 500 ppm
I20260620 15:17:14.125846 8853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:14 dist-test-slave-7s64 krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:14.128825 8853 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:17:14.130278 8853 webserver.cc:460] Webserver started at http://127.0.97.66:42061/ using document root <none> and password file <none>
I20260620 15:17:14.130872 8853 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:14.131012 8853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:14.131284 8853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:14.133991 8853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "95f479bbcec24be2ade31ad3b44493f9"
format_stamp: "Formatted at 2026-06-20 15:17:14 on dist-test-slave-7s64"
server_key: "e7391140cd5f3542bfae130a172b8052"
server_key_iv: "c3e56cfa108960e5581d65b688602b4e"
server_key_version: "encryptionkey@0"
I20260620 15:17:14.134783 8853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "95f479bbcec24be2ade31ad3b44493f9"
format_stamp: "Formatted at 2026-06-20 15:17:14 on dist-test-slave-7s64"
server_key: "e7391140cd5f3542bfae130a172b8052"
server_key_iv: "c3e56cfa108960e5581d65b688602b4e"
server_key_version: "encryptionkey@0"
I20260620 15:17:14.139570 8853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:17:14.142781 8868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:14.144001 8853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:17:14.144156 8853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "95f479bbcec24be2ade31ad3b44493f9"
format_stamp: "Formatted at 2026-06-20 15:17:14 on dist-test-slave-7s64"
server_key: "e7391140cd5f3542bfae130a172b8052"
server_key_iv: "c3e56cfa108960e5581d65b688602b4e"
server_key_version: "encryptionkey@0"
I20260620 15:17:14.144338 8853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:14.156291 8853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:14.160039 8853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:14.160303 8853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:14.161185 8853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:14.162371 8853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:14.162470 8853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:14.162619 8853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:14.162696 8853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:14.195549 8853 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:44519
I20260620 15:17:14.195629 8980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:44519 every 8 connection(s)
I20260620 15:17:14.196882 8853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:17:14.199826 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 8853
I20260620 15:17:14.200026 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:17:14.200590 389 external_mini_cluster.cc:1428] Setting key cd133b6ae7751f68958439203d01aa78
Jun 20 15:17:14 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:17:14.219211 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:14.199328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:33799 (local address 127.0.97.126:35025)
0620 15:17:14.199500 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:14.199524 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:14.200734 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 15:17:14.202849 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:14.202854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:14.202913 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:14.203024 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:14.205122 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.205807 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.208194 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.208639 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.208736 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:14.212435 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:14.212443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:14.212449 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:14.212516 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:14.216092 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:14.216806 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:14.216814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:14.216820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:14.216979 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:14.217364 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:14.217371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:14.217378 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:14.217742 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:14.217886 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:14.218493 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 15:17:14.218739 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:17:14.219703 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:14.199720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:33799)
0620 15:17:14.200570 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:14.200623 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:14.202562 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:14.203281 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:14.203295 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:14.204060 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:14.204939 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:14.204964 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.205961 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.205970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:14.208042 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:14.208059 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.208849 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.208859 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:14.209166 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:14.209857 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:14.209890 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:14.212230 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:14.216233 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:14.216239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:14.216250 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:14.216700 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:14.217082 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:14.217085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:14.217087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:14.217242 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:14.217854 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:14.217860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:14.218151 (+ 291us) client_negotiation.cc:769] Sending connection context
0620 15:17:14.218347 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 15:17:14.218672 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":340,"threads_started":1}
I20260620 15:17:14.221143 8981 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35025
I20260620 15:17:14.221473 8981 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:14.222342 8981 heartbeater.cc:511] Master 127.0.97.126:35025 requested a full tablet report, sending...
I20260620 15:17:14.223951 8658 ts_manager.cc:194] Registered new tserver with Master: 95f479bbcec24be2ade31ad3b44493f9 (127.0.97.66:44519)
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260620 15:17:14.224882 8658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:33799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:17:14.249847 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35025
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:14.560580 8986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:14.560824 8986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:14.560912 8986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:14.568152 8986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:14.568253 8986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:14.568367 8986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:17:14.578284 8986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:14.579574 8986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:14.581009 8986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:14.593412 8991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:14.593446 8992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:14.593511 8994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:14.594394 8986 server_base.cc:1034] running on GCE node
I20260620 15:17:14.594986 8986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:14.595849 8986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:14.597059 8986 hybrid_clock.cc:648] HybridClock initialized: now 1781968634597010 us; error 31 us; skew 500 ppm
I20260620 15:17:14.597349 8986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:14 dist-test-slave-7s64 krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:14.600376 8986 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:17:14.601876 8986 webserver.cc:460] Webserver started at http://127.0.97.67:38057/ using document root <none> and password file <none>
I20260620 15:17:14.602550 8986 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:14.602699 8986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:14.603017 8986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:14.605984 8986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "a5145beb3c8d41d19bc062814dddfe66"
format_stamp: "Formatted at 2026-06-20 15:17:14 on dist-test-slave-7s64"
server_key: "0b49321119d4d9089cbf036f87c7b64e"
server_key_iv: "28f3be2c0a37bff6ea9c85ca995eb9d6"
server_key_version: "encryptionkey@0"
I20260620 15:17:14.606794 8986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "a5145beb3c8d41d19bc062814dddfe66"
format_stamp: "Formatted at 2026-06-20 15:17:14 on dist-test-slave-7s64"
server_key: "0b49321119d4d9089cbf036f87c7b64e"
server_key_iv: "28f3be2c0a37bff6ea9c85ca995eb9d6"
server_key_version: "encryptionkey@0"
I20260620 15:17:14.611857 8986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:17:14.615370 9001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:14.616779 8986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:17:14.616940 8986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "a5145beb3c8d41d19bc062814dddfe66"
format_stamp: "Formatted at 2026-06-20 15:17:14 on dist-test-slave-7s64"
server_key: "0b49321119d4d9089cbf036f87c7b64e"
server_key_iv: "28f3be2c0a37bff6ea9c85ca995eb9d6"
server_key_version: "encryptionkey@0"
I20260620 15:17:14.617136 8986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:14.637625 8986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:14.641358 8986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:14.641624 8986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:14.642463 8986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:14.643741 8986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:14.643811 8986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:14.643975 8986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:14.644025 8986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:14.675405 8986 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:45083
I20260620 15:17:14.675480 9113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:45083 every 8 connection(s)
I20260620 15:17:14.676790 8986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:17:14.686733 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 8986
I20260620 15:17:14.686920 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:17:14.687405 389 external_mini_cluster.cc:1428] Setting key 2163183b33fef322b6952945aded9c64
Jun 20 15:17:14 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:17:14.695133 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:14.679327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55949 (local address 127.0.97.126:35025)
0620 15:17:14.679504 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:14.679509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:14.680705 (+ 1196us) server_negotiation.cc:407] Connection header received
0620 15:17:14.682427 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:14.682432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:14.682518 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:14.682629 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:14.683987 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.684824 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.686304 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.686940 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.687078 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:14.690487 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:14.690492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:14.690495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:14.690524 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:14.692776 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:14.693362 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:14.693365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:14.693367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:14.693472 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:14.693838 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:14.693841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:14.693843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:14.694086 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:14.694186 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:14.694736 (+ 550us) server_negotiation.cc:299] Negotiation successful
0620 15:17:14.694871 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:17:14.695935 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:14.679826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:55949)
0620 15:17:14.680560 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:14.680601 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:14.682178 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:14.682840 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:14.682848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:14.683305 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:14.683816 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:14.683836 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.684966 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.684977 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:14.686158 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:14.686169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.687163 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.687168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:14.687350 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:14.688008 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:14.688029 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:14.690324 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:14.692882 (+ 2558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:14.692888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:14.692897 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:14.693271 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:14.693565 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:14.693568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:14.693570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:14.693745 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:14.694182 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:14.694188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:14.694444 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 15:17:14.694658 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 15:17:14.694973 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":275,"threads_started":1}
I20260620 15:17:14.697294 9114 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35025
I20260620 15:17:14.697522 9114 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:14.698086 9114 heartbeater.cc:511] Master 127.0.97.126:35025 requested a full tablet report, sending...
I20260620 15:17:14.699317 8658 ts_manager.cc:194] Registered new tserver with Master: a5145beb3c8d41d19bc062814dddfe66 (127.0.97.67:45083)
I20260620 15:17:14.700111 8658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:55949
I20260620 15:17:14.702311 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab.
I20260620 15:17:14.754580 8847 heartbeater.cc:503] Master 127.0.97.126:35025 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab.
WARNING: no policy specified for oryx/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab.
I20260620 15:17:14.818768 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 8719
Jun 20 15:17:14 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:17:14.834438 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:14.820003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.0.97.65:39537)
0620 15:17:14.820741 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:14.820749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:14.820776 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:17:14.820832 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:14.820838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:14.821064 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:14.821194 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:14.822572 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.823521 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.824833 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.825386 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.825478 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:14.829430 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:14.829440 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:14.829469 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:14.829499 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:14.832072 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:14.832815 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:14.832822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:14.832824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:14.832956 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:14.833262 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:14.833265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:14.833267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:14.833559 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:14.833776 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:14.833980 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 15:17:14.834166 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":382,"threads_started":1}
I20260620 15:17:14.835670 8842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:14.917790 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 8719
I20260620 15:17:14.933941 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 8853
Jun 20 15:17:14 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:17:14.949347 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:14.934793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.0.97.66:44519)
0620 15:17:14.935294 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:14.935301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:14.935326 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:17:14.935381 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:14.935387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:14.935592 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:14.935727 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:14.936921 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.937899 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.939207 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:14.939749 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:14.939842 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:14.943996 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:14.944005 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:14.944017 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:14.944045 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:14.946753 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:14.947430 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:14.947436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:14.947438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:14.947553 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:14.948009 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:14.948012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:14.948014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:14.948301 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:14.948590 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:14.948911 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 15:17:14.949078 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
I20260620 15:17:14.950590 8976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:15.033007 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 8853
I20260620 15:17:15.049511 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 8986
Jun 20 15:17:15 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:17:15.066097 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:15.050517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.0.97.67:45083)
0620 15:17:15.051144 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:15.051152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:15.051179 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:17:15.051243 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:15.051249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:15.051461 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:15.051591 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:15.053016 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:15.054127 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:15.055492 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:15.056075 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:15.056185 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:15.060421 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:15.060432 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:15.060447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:15.060537 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:15.063579 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:15.064172 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:15.064179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:15.064181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:15.064312 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:15.064713 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:15.064717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:15.064719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:15.065061 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:15.065289 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:15.065505 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 15:17:15.065693 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":293,"threads_started":1}
I20260620 15:17:15.067325 9109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:15.154618 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 8986
I20260620 15:17:15.170781 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 8626
I20260620 15:17:15.171991 8688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:15.271063 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 8626
I20260620 15:17:15.291563 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35025
--webserver_interface=127.0.97.126
--webserver_port=33391
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:15.605049 9131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:15.605361 9131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:15.605516 9131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:15.612318 9131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:17:15.612447 9131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:15.612569 9131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:15.612664 9131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:17:15.612748 9131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:17:15.621969 9131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=33391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:15.623176 9131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:15.624542 9131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:15.636840 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:15.636754 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:15.636763 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:15.637601 9131 server_base.cc:1034] running on GCE node
I20260620 15:17:15.638324 9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:15.639698 9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:15.640966 9131 hybrid_clock.cc:648] HybridClock initialized: now 1781968635640898 us; error 42 us; skew 500 ppm
I20260620 15:17:15.641335 9131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:15 dist-test-slave-7s64 krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:15.644295 9131 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:17:15.645658 9131 webserver.cc:460] Webserver started at http://127.0.97.126:33391/ using document root <none> and password file <none>
I20260620 15:17:15.646246 9131 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:15.646356 9131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:15.651290 9131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 15:17:15.653860 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:15.655180 9131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:17:15.655313 9131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "b55563cf44124ffd8e43badb764b88bd"
format_stamp: "Formatted at 2026-06-20 15:17:13 on dist-test-slave-7s64"
server_key: "06332b73ea6816b79bd5c43fc2a40dcf"
server_key_iv: "17ef7e37c469141a70dfdf4cd8faa068"
server_key_version: "encryptionkey@0"
I20260620 15:17:15.656183 9131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:15.670524 9131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:15.674644 9131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:15.674844 9131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:15.692319 9131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35025
I20260620 15:17:15.692351 9197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35025 every 8 connection(s)
I20260620 15:17:15.693660 9131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:17:15.700006 9198 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:17:15.700846 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 9131
I20260620 15:17:15.701665 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:39537
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:35025
--webserver_port=41073
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.65
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:17:15.703300 9198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd: Bootstrap starting.
I20260620 15:17:15.721159 9198 log.cc:824] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:15.729200 9198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:17:15.729689 9198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd: Bootstrap complete.
I20260620 15:17:15.738241 9198 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } }
I20260620 15:17:15.739035 9198 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55563cf44124ffd8e43badb764b88bd, State: Initialized, Role: FOLLOWER
I20260620 15:17:15.739677 9198 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } }
I20260620 15:17:15.739908 9198 raft_consensus.cc:388] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:17:15.740021 9198 raft_consensus.cc:482] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:17:15.740211 9198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:17:15.743690 9198 raft_consensus.cc:504] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } }
I20260620 15:17:15.744151 9198 leader_election.cc:302] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b55563cf44124ffd8e43badb764b88bd; no voters:
I20260620 15:17:15.744771 9198 leader_election.cc:288] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:17:15.744891 9202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:17:15.746145 9202 raft_consensus.cc:686] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [term 2 LEADER]: Becoming Leader. State: Replica: b55563cf44124ffd8e43badb764b88bd, State: Running, Role: LEADER
I20260620 15:17:15.746718 9202 consensus_queue.cc:237] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } }
I20260620 15:17:15.746927 9198 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:17:15.750110 9204 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55563cf44124ffd8e43badb764b88bd. Latest consensus state: current_term: 2 leader_uuid: "b55563cf44124ffd8e43badb764b88bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } } }
I20260620 15:17:15.750530 9204 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:15.751061 9203 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b55563cf44124ffd8e43badb764b88bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55563cf44124ffd8e43badb764b88bd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35025 } } }
I20260620 15:17:15.751397 9203 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:15.752732 9208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:17:15.759302 9208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:17:15.761200 9208 catalog_manager.cc:1259] Loaded cluster ID: 0c5fca7b919648f2bc8fc63d02ce7476
I20260620 15:17:15.761349 9208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:17:15.766199 9208 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:17:15.768002 9208 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b55563cf44124ffd8e43badb764b88bd: Loaded TSK: 0
I20260620 15:17:15.769784 9208 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:17:16.049983 9200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:16.050259 9200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:16.050416 9200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:16.057617 9200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:16.057749 9200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:16.057919 9200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:17:16.067304 9200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:39537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=41073
--tserver_master_addrs=127.0.97.126:35025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:16.068573 9200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:16.069808 9200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:16.082675 9225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:16.082742 9224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:16.083032 9227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:16.084138 9200 server_base.cc:1034] running on GCE node
I20260620 15:17:16.084769 9200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:16.085624 9200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:16.086843 9200 hybrid_clock.cc:648] HybridClock initialized: now 1781968636086796 us; error 43 us; skew 500 ppm
I20260620 15:17:16.087138 9200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968636, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:16.090243 9200 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260620 15:17:16.091614 9200 webserver.cc:460] Webserver started at http://127.0.97.65:41073/ using document root <none> and password file <none>
I20260620 15:17:16.092248 9200 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:16.092394 9200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:16.097417 9200 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:17:16.100064 9234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:16.101488 9200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:17:16.101657 9200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "0ddf68add0364ed9a84b596b0a425102"
format_stamp: "Formatted at 2026-06-20 15:17:13 on dist-test-slave-7s64"
server_key: "0ae19eb8637ebab92252fb782ef80e61"
server_key_iv: "ef2fe25c8dc1def9ff22e0e133e8acd7"
server_key_version: "encryptionkey@0"
I20260620 15:17:16.102510 9200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:16.142266 9200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:16.146026 9200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:16.146298 9200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:16.147235 9200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:16.148478 9200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:16.148638 9200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:16.148775 9200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:16.148874 9200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:16.180688 9200 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:39537
I20260620 15:17:16.180737 9346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:39537 every 8 connection(s)
I20260620 15:17:16.182075 9200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:17:16.192134 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 9200
I20260620 15:17:16.192759 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:44519
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:35025
--webserver_port=42061
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.66
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:16.198983 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:16.184993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:46075)
0620 15:17:16.185801 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:16.185838 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:16.187408 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:16.188694 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:16.188706 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:16.189281 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:16.189782 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.189798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.191320 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.191325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.192638 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.192650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.193719 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.193725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.193969 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.194640 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:16.194661 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:16.198148 (+ 3487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":338,"threads_started":1}
W20260620 15:17:16.199623 9347 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35025 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.126:35025: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
W20260620 15:17:16.200347 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:16.184929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:46075 (local address 127.0.97.126:35025)
0620 15:17:16.185667 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:16.185682 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:16.185957 (+ 275us) server_negotiation.cc:407] Connection header received
0620 15:17:16.187905 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:16.187917 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:16.188211 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:16.188560 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:16.189978 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.191172 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.192774 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.193482 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.193689 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.198567 (+ 4878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:46075: BlockingRecv error: recv got EOF from 127.0.97.65:46075 (error 108)
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":252,"threads_started":1}
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:16.209211 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:16.200206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:49081)
0620 15:17:16.200597 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:16.200626 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:16.200718 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:16.201106 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:16.201114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:16.201528 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:16.201737 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.201750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.202715 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.202722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.204373 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.204390 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.205055 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.205063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.205285 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.205852 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:16.205878 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:16.208939 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120}
W20260620 15:17:16.209578 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:16.200450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49081 (local address 127.0.97.126:35025)
0620 15:17:16.200813 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:16.200819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:16.200836 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:17:16.200882 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:16.200887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:16.200950 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:16.201091 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:16.201886 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.202579 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.204558 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.204914 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.205004 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.209111 (+ 4107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:49081: BlockingRecv error: recv got EOF from 127.0.97.65:49081 (error 108)
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":952,"threads_started":1}
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:16.218511 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:16.209782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:46213)
0620 15:17:16.209977 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:16.209999 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:16.210102 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:16.210398 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:16.210406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:16.210826 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:16.211035 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.211049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.211963 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.211971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.213722 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.213740 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.214378 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.214386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.214638 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.215207 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:16.215235 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:16.218244 (+ 3009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 15:17:16.218665 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:16.210017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:46213 (local address 127.0.97.126:35025)
0620 15:17:16.210168 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:16.210171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:16.210184 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:17:16.210218 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:16.210221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:16.210268 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:16.210338 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:16.211176 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.211828 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.213856 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.214234 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.214323 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.218448 (+ 4125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:46213: BlockingRecv error: recv got EOF from 127.0.97.65:46213 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 15:17:16.218890 9347 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:17:16.505035 9351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:16.505364 9351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:16.505540 9351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:16.512817 9351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:16.512912 9351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:16.513077 9351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:17:16.522187 9351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:44519
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=42061
--tserver_master_addrs=127.0.97.126:35025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:16.523370 9351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:16.524593 9351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:16.536796 9358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:16.536755 9360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:16.536830 9357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:16.537925 9351 server_base.cc:1034] running on GCE node
I20260620 15:17:16.538534 9351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:16.539389 9351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:16.540601 9351 hybrid_clock.cc:648] HybridClock initialized: now 1781968636540544 us; error 45 us; skew 500 ppm
I20260620 15:17:16.540891 9351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968636, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:16.543758 9351 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260620 15:17:16.545250 9351 webserver.cc:460] Webserver started at http://127.0.97.66:42061/ using document root <none> and password file <none>
I20260620 15:17:16.545867 9351 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:16.546020 9351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:16.551190 9351 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:17:16.553746 9367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:16.555125 9351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:17:16.555269 9351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "95f479bbcec24be2ade31ad3b44493f9"
format_stamp: "Formatted at 2026-06-20 15:17:14 on dist-test-slave-7s64"
server_key: "e7391140cd5f3542bfae130a172b8052"
server_key_iv: "c3e56cfa108960e5581d65b688602b4e"
server_key_version: "encryptionkey@0"
I20260620 15:17:16.556173 9351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:16.572431 9351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:16.576004 9351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:16.576210 9351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:16.577040 9351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:16.578166 9351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:16.578244 9351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:16.578356 9351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:16.578429 9351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:16.610828 9351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:44519
I20260620 15:17:16.610885 9479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:44519 every 8 connection(s)
I20260620 15:17:16.612180 9351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:17:16.614173 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 9351
I20260620 15:17:16.614833 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:45083
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:35025
--webserver_port=38057
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.67
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:17:16.630775 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:16.614701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41083 (local address 127.0.97.126:35025)
0620 15:17:16.614932 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:16.614938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:16.616193 (+ 1255us) server_negotiation.cc:407] Connection header received
0620 15:17:16.618973 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:16.618977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:16.619037 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:16.619122 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:16.621150 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.621969 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.623946 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.624346 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.624436 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.630535 (+ 6099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:41083: BlockingRecv error: recv got EOF from 127.0.97.66:41083 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:17:16.631583 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:16.615240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:41083)
0620 15:17:16.616022 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:16.616070 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:16.618672 (+ 2602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:16.619370 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:16.619383 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:16.620159 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:16.620963 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.620989 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.622132 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.622136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.623797 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.623814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.624591 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.624601 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.624915 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.625608 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:16.625639 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:16.630337 (+ 4698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":306,"threads_started":1}
W20260620 15:17:16.632340 9480 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35025 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.126:35025: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:16.641834 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:16.632944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:35787)
0620 15:17:16.633133 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:16.633153 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:16.633254 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:16.633553 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:16.633559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:16.634010 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:16.634219 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.634233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.635144 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.635153 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.636891 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.636907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.637533 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.637542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.637762 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.638312 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:16.638340 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:16.641585 (+ 3245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 15:17:16.642028 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:16.633044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:35787 (local address 127.0.97.126:35025)
0620 15:17:16.633187 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:16.633190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:16.633205 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:17:16.633364 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:16.633368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:16.633419 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:16.633493 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:16.634360 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.635013 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.637022 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.637389 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.637477 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.641808 (+ 4331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:35787: BlockingRecv error: recv got EOF from 127.0.97.66:35787 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:16.650228 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:16.642625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:36539)
0620 15:17:16.642786 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:16.642810 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:16.642880 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:16.643136 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:16.643141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:16.643374 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:16.643497 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.643503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.644798 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.644802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.645667 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:16.645677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.646596 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.646599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:16.646739 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.647325 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:16.647341 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:16.649967 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:17:16.650244 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:16.642717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:36539 (local address 127.0.97.126:35025)
0620 15:17:16.642859 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:16.642864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:16.642882 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:17:16.642954 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:16.642959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:16.643021 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:16.643112 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:16.643612 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.644682 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.645784 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:16.646465 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:16.646594 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:16.650110 (+ 3516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:36539: BlockingRecv error: recv got EOF from 127.0.97.66:36539 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 15:17:16.650519 9480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:17:16.934687 9483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:16.934966 9483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:16.935142 9483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:16.941941 9483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:16.942076 9483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:16.942271 9483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:17:16.951524 9483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:45083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=38057
--tserver_master_addrs=127.0.97.126:35025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:16.952708 9483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:16.953989 9483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:16.966413 9489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:16.966446 9491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:16.966606 9488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:16.967370 9483 server_base.cc:1034] running on GCE node
I20260620 15:17:16.967983 9483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:16.968914 9483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:16.970131 9483 hybrid_clock.cc:648] HybridClock initialized: now 1781968636970095 us; error 48 us; skew 500 ppm
I20260620 15:17:16.970427 9483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:16 dist-test-slave-7s64 krb5kdc[8613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968636, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:16.973459 9483 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260620 15:17:16.974673 9483 webserver.cc:460] Webserver started at http://127.0.97.67:38057/ using document root <none> and password file <none>
I20260620 15:17:16.975251 9483 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:16.975351 9483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:16.980382 9483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:17:16.982862 9498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:16.984084 9483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:17:16.984251 9483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "a5145beb3c8d41d19bc062814dddfe66"
format_stamp: "Formatted at 2026-06-20 15:17:14 on dist-test-slave-7s64"
server_key: "0b49321119d4d9089cbf036f87c7b64e"
server_key_iv: "28f3be2c0a37bff6ea9c85ca995eb9d6"
server_key_version: "encryptionkey@0"
I20260620 15:17:16.985219 9483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:17.010205 9483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:17.014390 9483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:17.014668 9483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:17.015642 9483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:17.016935 9483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:17.017019 9483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:17.017164 9483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:17.017257 9483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:17.049950 9483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:45083
I20260620 15:17:17.050001 9610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:45083 every 8 connection(s)
I20260620 15:17:17.051261 9483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:17:17.058352 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 9483
Jun 20 15:17:17 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:17 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:17:17.068624 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:17.053665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:52867 (local address 127.0.97.126:35025)
0620 15:17:17.053861 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:17.053867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:17.055124 (+ 1257us) server_negotiation.cc:407] Connection header received
0620 15:17:17.057166 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:17.057170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:17.057234 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:17.057311 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:17.059314 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.060666 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.062272 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.062781 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.062941 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.068258 (+ 5317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:52867: BlockingRecv error: recv got EOF from 127.0.97.67:52867 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260620 15:17:17.069211 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:17.054193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:52867)
0620 15:17:17.054902 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:17.055008 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:17.056870 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:17.057526 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:17.057534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:17.058241 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:17.059059 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:17.059112 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.060816 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.060821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:17.062118 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:17.062131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.063010 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.063018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:17.063248 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.063962 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:17.063985 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:17.068009 (+ 4024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":269,"threads_started":1}
W20260620 15:17:17.069856 9611 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35025 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.126:35025: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
I20260620 15:17:17.074230 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:17.059453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.0.97.126:35025)
0620 15:17:17.060170 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:17.060175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:17.060188 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:17:17.060227 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:17.060230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:17.060310 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:17.060434 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:17.061920 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.062654 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.064034 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.064429 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.064579 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.067394 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:17.067402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:17.067415 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:17.067446 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:17.071008 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:17.071808 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:17.071816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:17.071823 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:17.072045 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:17.072385 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:17.072390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:17.072393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:17.072957 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:17.073182 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:17.073608 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 15:17:17.073844 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":518,"threads_started":1}
Jun 20 15:17:17 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:17 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:17.079562 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:17.070436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:51935)
0620 15:17:17.070615 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:17.070642 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:17.070720 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:17.071022 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:17.071026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:17.071477 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:17.071664 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:17.071674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.072977 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.072981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:17.074453 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:17.074467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.075377 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.075381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:17.075597 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.076114 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:17.076136 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:17.079307 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:17:17.079689 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:17.070552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51935 (local address 127.0.97.126:35025)
0620 15:17:17.070714 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:17.070719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:17.070739 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:17:17.070791 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:17.070796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:17.070867 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:17.070966 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:17.071801 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.072852 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.074609 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.075207 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.075332 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.079467 (+ 4135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:51935: BlockingRecv error: recv got EOF from 127.0.97.67:51935 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 15:17:17 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:17 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:17.086866 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:17.080335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:50591)
0620 15:17:17.080539 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:17.080558 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:17.080642 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:17.080866 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:17.080869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:17.081103 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:17.081238 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:17.081244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.082149 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.082153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:17.082985 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:17.082994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.083649 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.083653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:17.083759 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.084276 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:17.084292 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:17.086691 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 15:17:17.087109 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:17.080412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:50591 (local address 127.0.97.126:35025)
0620 15:17:17.080647 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:17.080651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:17.080665 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:17:17.080702 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:17.080705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:17.080761 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:17.080833 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:17.081349 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.082030 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.083124 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.083526 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.083633 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.086844 (+ 3211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:50591: BlockingRecv error: recv got EOF from 127.0.97.67:50591 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
W20260620 15:17:17.087152 9611 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 15:17:17 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:17 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:17.227644 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:17.219403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:43129)
0620 15:17:17.219957 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:17.219977 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:17.220126 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:17.220475 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:17.220479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:17.220971 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:17.221150 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:17.221159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.222157 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.222163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:17.223114 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:17.223124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.223850 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.223856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:17.224033 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.224619 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:17.224636 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:17.227424 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":266,"threads_started":1}
W20260620 15:17:17.227815 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:17.219462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:43129 (local address 127.0.97.126:35025)
0620 15:17:17.219689 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:17.219696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:17.220066 (+ 370us) server_negotiation.cc:407] Connection header received
0620 15:17:17.220262 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:17.220268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:17.220344 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:17.220444 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:17.221292 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.222016 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.223253 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.223668 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.223794 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.227608 (+ 3814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:43129: BlockingRecv error: recv got EOF from 127.0.97.65:43129 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 15:17:17 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:17 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:17.659865 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:17.651056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:36811)
0620 15:17:17.651566 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:17.651582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:17.651687 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:17.651951 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:17.651954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:17.652401 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:17.652669 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:17.652679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.653764 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.653768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:17.655082 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:17.655093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.655877 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.655881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:17.656027 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.656665 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:17.656683 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:17.659545 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":272,"threads_started":1}
W20260620 15:17:17.660012 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:17.651291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:36811 (local address 127.0.97.126:35025)
0620 15:17:17.651496 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:17.651501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:17.651648 (+ 147us) server_negotiation.cc:407] Connection header received
0620 15:17:17.651775 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:17.651779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:17.651835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:17.651915 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:17.652798 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.653610 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.655235 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:17.655733 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:17.655829 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:17.659757 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:36811: BlockingRecv error: recv got EOF from 127.0.97.66:36811 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 15:17:18 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:18 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:18.098325 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:18.087797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:51697)
0620 15:17:18.088339 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:18.088376 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:18.088578 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:18.089078 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:18.089083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:18.089673 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:18.089921 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:18.089931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.091480 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.091487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:18.093359 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:18.093372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.094548 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.094553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:18.094729 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:18.095352 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:18.095383 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:18.098114 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":297,"threads_started":1}
W20260620 15:17:18.098691 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:18.087806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51697 (local address 127.0.97.126:35025)
0620 15:17:18.087989 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:18.087995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:18.088449 (+ 454us) server_negotiation.cc:407] Connection header received
0620 15:17:18.088801 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:18.088809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:18.088906 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:18.089022 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:18.090059 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.091318 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.093593 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.094386 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.094532 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:18.098381 (+ 3849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:51697: BlockingRecv error: recv got EOF from 127.0.97.67:51697 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 15:17:18 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:18 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:18.236878 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:18.228702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:55153)
0620 15:17:18.229220 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:18.229236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:18.229380 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:18.229744 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:18.229749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:18.230146 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:18.230281 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:18.230289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.231309 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.231317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:18.232223 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:18.232234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.232984 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.232989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:18.233151 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:18.233890 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:18.233908 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:18.236651 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":295,"threads_started":1}
W20260620 15:17:18.237039 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:18.228740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:55153 (local address 127.0.97.126:35025)
0620 15:17:18.228932 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:18.228938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:18.229321 (+ 383us) server_negotiation.cc:407] Connection header received
0620 15:17:18.229525 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:18.229531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:18.229601 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:18.229704 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:18.230421 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.231185 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.232364 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.232833 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.232970 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:18.236826 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:55153: BlockingRecv error: recv got EOF from 127.0.97.65:55153 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 15:17:18 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:18 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:18.668835 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:18.660983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:40171)
0620 15:17:18.661510 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:18.661546 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:18.661678 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:18.662059 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:18.662064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:18.662581 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:18.662715 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:18.662723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.663650 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.663654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:18.664549 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:18.664559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.665283 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.665287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:18.665392 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:18.665915 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:18.665930 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:18.668605 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":277,"threads_started":1}
W20260620 15:17:18.668970 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:18.661126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:40171 (local address 127.0.97.126:35025)
0620 15:17:18.661333 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:18.661338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:18.661627 (+ 289us) server_negotiation.cc:407] Connection header received
0620 15:17:18.661806 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:18.661811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:18.661895 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:18.662000 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:18.662848 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.663549 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.664689 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:18.665144 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:18.665258 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:18.668764 (+ 3506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:40171: BlockingRecv error: recv got EOF from 127.0.97.66:40171 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 15:17:19 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:19 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:19.107859 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:19.099564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:35373)
0620 15:17:19.100088 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:19.100104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:19.100258 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:19.100637 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:19.100641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:19.101062 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:19.101196 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:19.101204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.102198 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.102206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:19.103093 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:19.103104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.103792 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.103797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:19.103960 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:19.104507 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:19.104524 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:19.107634 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":289,"threads_started":1}
W20260620 15:17:19.108045 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:19.099727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35373 (local address 127.0.97.126:35025)
0620 15:17:19.099943 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:19.099949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:19.100189 (+ 240us) server_negotiation.cc:407] Connection header received
0620 15:17:19.100376 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:19.100383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:19.100456 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:19.100602 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:19.101343 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.102054 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.103229 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.103645 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.103781 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:19.107800 (+ 4019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:35373: BlockingRecv error: recv got EOF from 127.0.97.67:35373 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 15:17:19 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:19 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:19.246402 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:19.237851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:55123)
0620 15:17:19.238347 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:19.238362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:19.238521 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:19.238856 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:19.238861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:19.239213 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:19.239368 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:19.239376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.240383 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.240391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:19.241317 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:19.241328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.241991 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.241996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:19.242170 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:19.242868 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:19.242886 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:19.246050 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":257,"threads_started":1}
W20260620 15:17:19.246502 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:19.237870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:55123 (local address 127.0.97.126:35025)
0620 15:17:19.238070 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:19.238079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:19.238443 (+ 364us) server_negotiation.cc:407] Connection header received
0620 15:17:19.238651 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:19.238657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:19.238727 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:19.238824 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:19.239510 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.240252 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.241450 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.241851 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.241991 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:19.246236 (+ 4245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:55123: BlockingRecv error: recv got EOF from 127.0.97.65:55123 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 15:17:19 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:19 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:19.678189 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:19.669802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:57267)
0620 15:17:19.670279 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:19.670295 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:19.670430 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:19.670739 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:19.670742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:19.671066 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:19.671244 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:19.671251 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.672364 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.672371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:19.673633 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:19.673647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.674329 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.674335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:19.674511 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:19.675092 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:19.675108 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:19.677829 (+ 2721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":247,"threads_started":1}
W20260620 15:17:19.678241 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:19.669874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:57267 (local address 127.0.97.126:35025)
0620 15:17:19.670082 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:19.670088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:19.670370 (+ 282us) server_negotiation.cc:407] Connection header received
0620 15:17:19.670550 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:19.670553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:19.670613 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:19.670690 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:19.671358 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.672234 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.673772 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:19.674189 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:19.674318 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:19.678020 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:57267: BlockingRecv error: recv got EOF from 127.0.97.66:57267 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 15:17:20 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:20 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:20.116966 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:20.108879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:54523)
0620 15:17:20.109439 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:20.109458 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:20.109604 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:20.110003 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:20.110008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:20.110427 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:20.110599 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:20.110608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.111655 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.111658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:20.112592 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:20.112602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.113275 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.113279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:20.113397 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:20.113984 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:20.114003 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:20.116732 (+ 2729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":281,"threads_started":1}
W20260620 15:17:20.117141 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:20.108974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54523 (local address 127.0.97.126:35025)
0620 15:17:20.109160 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:20.109168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:20.109561 (+ 393us) server_negotiation.cc:407] Connection header received
0620 15:17:20.109714 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:20.109721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:20.109851 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:20.109963 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:20.110761 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.111521 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.112727 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.113148 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.113262 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:20.116911 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:54523: BlockingRecv error: recv got EOF from 127.0.97.67:54523 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 15:17:20 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:20 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:20.256242 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:20.247547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:52861)
0620 15:17:20.248206 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:20.248224 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:20.248348 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:20.248766 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:20.248772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:20.249345 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:20.249531 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:20.249542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.250687 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.250693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:20.251681 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:20.251694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.252395 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.252401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:20.252636 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:20.253206 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:20.253224 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:20.255972 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":269,"threads_started":1}
W20260620 15:17:20.256438 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:20.247584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:52861 (local address 127.0.97.126:35025)
0620 15:17:20.247806 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:20.247815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:20.248303 (+ 488us) server_negotiation.cc:407] Connection header received
0620 15:17:20.248550 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:20.248555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:20.248618 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:20.248735 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:20.249675 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.250545 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.251833 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.252247 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.252401 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:20.256163 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:52861: BlockingRecv error: recv got EOF from 127.0.97.65:52861 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 15:17:20 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:20 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:20.687314 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:20.679069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:42343)
0620 15:17:20.679552 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:20.679568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:20.679687 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:20.679958 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:20.679961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:20.680358 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:20.680558 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:20.680573 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.681477 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.681481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:20.682434 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:20.682444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.683084 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.683088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:20.683209 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:20.684056 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:20.684081 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:20.687085 (+ 3004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":281,"threads_started":1}
W20260620 15:17:20.687395 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:20.679114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:42343 (local address 127.0.97.126:35025)
0620 15:17:20.679285 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:20.679289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:20.679648 (+ 359us) server_negotiation.cc:407] Connection header received
0620 15:17:20.679780 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:20.679783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:20.679840 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:20.679917 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:20.680678 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.681358 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.682549 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:20.682949 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:20.683037 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:20.687235 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:42343: BlockingRecv error: recv got EOF from 127.0.97.66:42343 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 15:17:21 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:21 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:21.125994 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:21.117827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:54129)
0620 15:17:21.118273 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:21.118288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:21.118436 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:21.118816 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:21.118822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:21.119312 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:21.119518 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:21.119530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.120749 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.120753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:21.121712 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:21.121723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.122392 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.122396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:21.122528 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:21.123144 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:21.123161 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:21.125774 (+ 2613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":235,"threads_started":1}
W20260620 15:17:21.126231 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:21.117894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54129 (local address 127.0.97.126:35025)
0620 15:17:21.118139 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:21.118145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:21.118369 (+ 224us) server_negotiation.cc:407] Connection header received
0620 15:17:21.118551 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:21.118558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:21.118650 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:21.118751 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:21.119708 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.120608 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.121831 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.122244 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.122346 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:21.125951 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:54129: BlockingRecv error: recv got EOF from 127.0.97.67:54129 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 15:17:21 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:21 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:21.265421 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:21.257154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:37157)
0620 15:17:21.257586 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:21.257602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:21.257707 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:21.258055 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:21.258059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:21.258427 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:21.258580 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:21.258588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.259759 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.259763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:21.261199 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:21.261212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.261870 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.261874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:21.261984 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:21.262527 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:21.262544 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:21.265190 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":222,"threads_started":1}
W20260620 15:17:21.265769 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:21.257241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:37157 (local address 127.0.97.126:35025)
0620 15:17:21.257483 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:21.257488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:21.257682 (+ 194us) server_negotiation.cc:407] Connection header received
0620 15:17:21.257843 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:21.257847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:21.257907 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:21.257990 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:21.258715 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.259611 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.261313 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.261736 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.261834 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:21.265502 (+ 3668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:37157: BlockingRecv error: recv got EOF from 127.0.97.65:37157 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 20 15:17:21 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:21 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:21.696750 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:21.688238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:60831)
0620 15:17:21.688774 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:21.688790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:21.688892 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:21.689204 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:21.689208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:21.689676 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:21.689822 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:21.689830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.690777 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.690780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:21.691650 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:21.691660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.692425 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.692430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:21.692648 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:21.693291 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:21.693308 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:21.696397 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":312,"threads_started":1}
W20260620 15:17:21.697108 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:21.688523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:60831 (local address 127.0.97.126:35025)
0620 15:17:21.688697 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:21.688704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:21.688913 (+ 209us) server_negotiation.cc:407] Connection header received
0620 15:17:21.688969 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:21.688973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:21.689029 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:21.689136 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:21.689949 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.690654 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.691794 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:21.692274 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:21.692418 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:21.696645 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:60831: BlockingRecv error: recv got EOF from 127.0.97.66:60831 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 15:17:22 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:22 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:22.135169 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:22.127016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:34617)
0620 15:17:22.127570 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:22.127586 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:22.127742 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:22.128099 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:22.128106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:22.128470 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:22.128681 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:22.128689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.129732 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.129740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:22.130639 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:22.130649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.131320 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.131325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:22.131492 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:22.132081 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:22.132097 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:22.134755 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":287,"threads_started":1}
W20260620 15:17:22.135192 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:22.127132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:34617 (local address 127.0.97.126:35025)
0620 15:17:22.127367 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:22.127374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:22.127674 (+ 300us) server_negotiation.cc:407] Connection header received
0620 15:17:22.127874 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:22.127879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:22.127948 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:22.128051 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:22.128818 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.129597 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.130776 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.131182 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.131311 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:22.134955 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:34617: BlockingRecv error: recv got EOF from 127.0.97.67:34617 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 15:17:22 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:22 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:22.274390 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:22.266332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:37981)
0620 15:17:22.266794 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:22.266810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:22.266920 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:22.267252 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:22.267255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:22.267596 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:22.267751 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:22.267759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.268876 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.268881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:22.269962 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:22.269974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.270699 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.270705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:22.270869 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:22.271406 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:22.271422 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:22.274083 (+ 2661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":268,"threads_started":1}
W20260620 15:17:22.274506 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:22.266436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:37981 (local address 127.0.97.126:35025)
0620 15:17:22.266685 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:22.266691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:22.266888 (+ 197us) server_negotiation.cc:407] Connection header received
0620 15:17:22.267063 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:22.267066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:22.267128 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:22.267212 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:22.267868 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.268722 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.270099 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.270522 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.270655 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:22.274271 (+ 3616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:37981: BlockingRecv error: recv got EOF from 127.0.97.65:37981 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Jun 20 15:17:22 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:22 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:22.706041 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:22.697664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:41483)
0620 15:17:22.698170 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:22.698186 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:22.698343 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:22.698669 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:22.698673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:22.699079 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:22.699216 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:22.699224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.700250 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.700256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:22.701147 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:22.701158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.701931 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.701941 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:22.702137 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:22.702844 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:22.702885 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:22.705748 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":250,"threads_started":1}
W20260620 15:17:22.706247 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:22.697733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41483 (local address 127.0.97.126:35025)
0620 15:17:22.697926 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:22.697933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:22.698276 (+ 343us) server_negotiation.cc:407] Connection header received
0620 15:17:22.698467 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:22.698476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:22.698543 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:22.698645 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:22.699383 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.700106 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.701288 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:22.701760 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:22.701889 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:22.706024 (+ 4135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:41483: BlockingRecv error: recv got EOF from 127.0.97.66:41483 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 15:17:23 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:23 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:23.149114 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:23.136156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:55501)
0620 15:17:23.136851 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:23.136867 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:23.137014 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:23.137382 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:23.137387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:23.137906 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:23.138100 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:23.138111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.139654 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.139659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:23.140956 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:23.140967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.141890 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.141894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:23.142020 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:23.145982 (+ 3962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:23.145999 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:23.148904 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":464,"threads_started":1}
W20260620 15:17:23.149346 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:23.136201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55501 (local address 127.0.97.126:35025)
0620 15:17:23.136404 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:23.136414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:23.136968 (+ 554us) server_negotiation.cc:407] Connection header received
0620 15:17:23.137157 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:23.137162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:23.137241 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:23.137351 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:23.138255 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.139491 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.141096 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.141746 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.141844 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:23.149062 (+ 7218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:55501: BlockingRecv error: recv got EOF from 127.0.97.67:55501 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 15:17:23 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:23 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:23.282899 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:23.275316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:56147)
0620 15:17:23.275869 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:23.275885 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:23.275992 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:23.276281 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:23.276284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:23.276680 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:23.276844 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:23.276852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.277766 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.277769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:23.278689 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:23.278700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.279346 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.279351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:23.279454 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:23.280012 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:23.280029 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:23.282671 (+ 2642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":276,"threads_started":1}
W20260620 15:17:23.283025 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:23.275372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56147 (local address 127.0.97.126:35025)
0620 15:17:23.275556 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:23.275560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:23.275961 (+ 401us) server_negotiation.cc:407] Connection header received
0620 15:17:23.276095 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:23.276098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:23.276158 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:23.276236 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:23.276956 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.277647 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.278817 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.279221 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.279314 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:23.282837 (+ 3523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:56147: BlockingRecv error: recv got EOF from 127.0.97.65:56147 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 15:17:23 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:23 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:23.716284 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:23.707175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:39797)
0620 15:17:23.707933 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:23.707956 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:23.708138 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:23.708578 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:23.708583 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:23.709075 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:23.709254 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:23.709265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.710412 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.710419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:23.711482 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:23.711495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.712462 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.712467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:23.712695 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:23.713256 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:23.713274 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:23.716056 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":355,"threads_started":1}
W20260620 15:17:23.716477 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:23.707190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39797 (local address 127.0.97.126:35025)
0620 15:17:23.707432 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:23.707488 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:23.708056 (+ 568us) server_negotiation.cc:407] Connection header received
0620 15:17:23.708284 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:23.708291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:23.708390 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:23.708543 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:23.709413 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.710256 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.711677 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:23.712192 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:23.712454 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:23.716224 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:39797: BlockingRecv error: recv got EOF from 127.0.97.66:39797 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Jun 20 15:17:24 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:24 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:24.158664 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:24.149998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:50985)
0620 15:17:24.150545 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:24.150560 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:24.150685 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:24.151009 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:24.151013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:24.151412 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:24.151593 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:24.151602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.152654 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.152658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:24.153679 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:24.153690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.154528 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.154533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:24.154652 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:24.155261 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:24.155279 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:24.158284 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":266,"threads_started":1}
W20260620 15:17:24.158686 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:24.150083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:50985 (local address 127.0.97.126:35025)
0620 15:17:24.150313 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:24.150323 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:24.150649 (+ 326us) server_negotiation.cc:407] Connection header received
0620 15:17:24.150784 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:24.150788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:24.150868 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:24.150982 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:24.151730 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.152524 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.153809 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.154385 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.154503 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:24.158470 (+ 3967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:50985: BlockingRecv error: recv got EOF from 127.0.97.67:50985 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 15:17:24 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:24 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:24.292416 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:24.283786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:39405)
0620 15:17:24.284296 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:24.284312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:24.284464 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:24.284865 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:24.284870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:24.285440 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:24.285652 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:24.285665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.286826 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.286832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:24.287729 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:24.287740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.288599 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.288604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:24.288725 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:24.289357 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:24.289374 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:24.292086 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":280,"threads_started":1}
W20260620 15:17:24.292551 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:24.283986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:39405 (local address 127.0.97.126:35025)
0620 15:17:24.284192 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:24.284199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:24.284390 (+ 191us) server_negotiation.cc:407] Connection header received
0620 15:17:24.284648 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:24.284654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:24.284728 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:24.284836 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:24.285797 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.286688 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.287863 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.288364 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.288518 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:24.292295 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:39405: BlockingRecv error: recv got EOF from 127.0.97.65:39405 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 15:17:24 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:24 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:24.725654 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:24.717206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:35435)
0620 15:17:24.717727 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:24.717743 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:24.717853 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:24.718141 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:24.718144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:24.718524 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:24.718662 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:24.718670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.719825 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.719829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:24.720754 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:24.720765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.721673 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.721680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:24.721854 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:24.722543 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:24.722563 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:24.725334 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
W20260620 15:17:24.725737 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:24.717293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:35435 (local address 127.0.97.126:35025)
0620 15:17:24.717464 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:24.717468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:24.717812 (+ 344us) server_negotiation.cc:407] Connection header received
0620 15:17:24.717943 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:24.717946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:24.718003 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:24.718112 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:24.718775 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.719676 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.720886 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:24.721485 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:24.721628 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:24.725515 (+ 3887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:35435: BlockingRecv error: recv got EOF from 127.0.97.66:35435 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 15:17:25 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:25 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:25.167634 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:25.159648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:40789)
0620 15:17:25.160230 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:25.160245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:25.160350 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:25.160759 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:25.160763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:25.161110 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:25.161247 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:25.161255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.162306 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.162310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:25.163179 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:25.163188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.163891 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.163895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:25.164003 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:25.164683 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:25.164700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:25.167424 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":284,"threads_started":1}
W20260620 15:17:25.167867 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:25.159679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:40789 (local address 127.0.97.126:35025)
0620 15:17:25.159854 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:25.159858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:25.160301 (+ 443us) server_negotiation.cc:407] Connection header received
0620 15:17:25.160446 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:25.160450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:25.160612 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:25.160711 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:25.161366 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.162176 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.163294 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.163738 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.163835 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:25.167609 (+ 3774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:40789: BlockingRecv error: recv got EOF from 127.0.97.67:40789 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 15:17:25 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:25 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:25.301192 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:25.293414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:60427)
0620 15:17:25.293939 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:25.293960 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:25.294108 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:25.294530 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:25.294536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:25.294897 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:25.295054 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:25.295063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.296015 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.296019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:25.296952 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:25.296963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.297636 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.297640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:25.297757 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:25.298283 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:25.298300 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:25.300952 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":290,"threads_started":1}
W20260620 15:17:25.301311 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:25.293476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:60427 (local address 127.0.97.126:35025)
0620 15:17:25.293655 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:25.293662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:25.294091 (+ 429us) server_negotiation.cc:407] Connection header received
0620 15:17:25.294260 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:25.294267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:25.294364 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:25.294478 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:25.295177 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.295909 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.297077 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.297516 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.297606 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:25.301126 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:60427: BlockingRecv error: recv got EOF from 127.0.97.65:60427 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 15:17:25 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:25 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:25.734877 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:25.726739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:56561)
0620 15:17:25.727198 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:25.727214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:25.727322 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:25.727612 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:25.727616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:25.727963 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:25.728118 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:25.728126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.729287 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.729291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:25.730280 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:25.730293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.730996 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.731001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:25.731130 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:25.731735 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:25.731754 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:25.734614 (+ 2860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":226,"threads_started":1}
W20260620 15:17:25.735118 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:25.726713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56561 (local address 127.0.97.126:35025)
0620 15:17:25.726917 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:25.726926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:25.727292 (+ 366us) server_negotiation.cc:407] Connection header received
0620 15:17:25.727414 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:25.727418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:25.727479 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:25.727560 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:25.728253 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.729156 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.730430 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:25.730848 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:25.730949 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:25.734825 (+ 3876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:56561: BlockingRecv error: recv got EOF from 127.0.97.66:56561 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 15:17:26 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:26 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:26.177068 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:26.168535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:46119)
0620 15:17:26.169005 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:26.169022 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:26.169170 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:26.169506 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:26.169511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:26.169901 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:26.170059 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:26.170078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.171338 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.171344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:26.172673 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:26.172686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.173331 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.173336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:26.173450 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:26.174103 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:26.174121 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:26.176830 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":257,"threads_started":1}
W20260620 15:17:26.177278 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:26.168603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:46119 (local address 127.0.97.126:35025)
0620 15:17:26.168828 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:26.168834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:26.169139 (+ 305us) server_negotiation.cc:407] Connection header received
0620 15:17:26.169268 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:26.169271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:26.169343 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:26.169422 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:26.170221 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.171174 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.172790 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.173193 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.173296 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:26.177030 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:46119: BlockingRecv error: recv got EOF from 127.0.97.67:46119 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 15:17:26 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:26 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:26.310245 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:26.302182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:35831)
0620 15:17:26.302682 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:26.302699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:26.302805 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:26.303083 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:26.303086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:26.303420 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:26.303558 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:26.303566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.304642 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.304647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:26.305620 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:26.305631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.306337 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.306341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:26.306457 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:26.307115 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:26.307132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:26.309872 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":291,"threads_started":1}
W20260620 15:17:26.310271 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:26.302208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:35831 (local address 127.0.97.126:35025)
0620 15:17:26.302389 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:26.302393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:26.302766 (+ 373us) server_negotiation.cc:407] Connection header received
0620 15:17:26.302900 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:26.302903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:26.302960 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:26.303037 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:26.303696 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.304514 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.305762 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.306211 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.306306 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:26.310045 (+ 3739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:35831: BlockingRecv error: recv got EOF from 127.0.97.65:35831 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 15:17:26 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:26 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:26.743427 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:26.735735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:58791)
0620 15:17:26.736281 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:26.736298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:26.736407 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:26.736714 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:26.736717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:26.737053 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:26.737186 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:26.737194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.738213 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.738217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:26.739160 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:26.739171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.739810 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.739815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:26.739935 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:26.740620 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:26.740637 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:26.743171 (+ 2534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":305,"threads_started":1}
W20260620 15:17:26.743629 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:26.735809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:58791 (local address 127.0.97.126:35025)
0620 15:17:26.735996 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:26.736000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:26.736382 (+ 382us) server_negotiation.cc:407] Connection header received
0620 15:17:26.736511 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:26.736514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:26.736591 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:26.736670 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:26.737326 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.738069 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.739273 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:26.739667 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:26.739773 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:26.743376 (+ 3603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:58791: BlockingRecv error: recv got EOF from 127.0.97.66:58791 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 15:17:27 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:27 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:27.187405 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:27.178064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:46697)
0620 15:17:27.178653 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:27.178670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:27.178854 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:27.179235 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:27.179240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:27.180183 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:27.180376 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:27.180385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.181478 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.181487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:27.182460 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:27.182472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.183217 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.183223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:27.183436 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:27.184119 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:27.184144 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:27.187033 (+ 2889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":358,"threads_started":1}
W20260620 15:17:27.187532 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:27.178268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:46697 (local address 127.0.97.126:35025)
0620 15:17:27.178475 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:27.178481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:27.178785 (+ 304us) server_negotiation.cc:407] Connection header received
0620 15:17:27.178985 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:27.178993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:27.179077 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:27.179205 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:27.180560 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.181321 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.182625 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.183057 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.183206 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:27.187235 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:46697: BlockingRecv error: recv got EOF from 127.0.97.67:46697 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 15:17:27 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:27 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:27.319094 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:27.311086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:56257)
0620 15:17:27.311514 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:27.311530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:27.311638 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:27.311903 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:27.311906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:27.312298 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:27.312460 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:27.312468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.313441 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.313445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:27.314324 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:27.314334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.315062 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.315068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:27.315196 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:27.315766 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:27.315785 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:27.318738 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":230,"threads_started":1}
W20260620 15:17:27.319150 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:27.311178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56257 (local address 127.0.97.126:35025)
0620 15:17:27.311401 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:27.311407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:27.311604 (+ 197us) server_negotiation.cc:407] Connection header received
0620 15:17:27.311747 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:27.311750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:27.311811 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:27.311889 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:27.312604 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.313330 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.314446 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.314873 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.314968 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:27.318934 (+ 3966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:56257: BlockingRecv error: recv got EOF from 127.0.97.65:56257 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 15:17:27 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:27 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:27.752177 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:27.744281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:60047)
0620 15:17:27.744751 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:27.744767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:27.744869 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:27.745161 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:27.745164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:27.745524 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:27.745703 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:27.745711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.746719 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.746722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:27.747655 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:27.747666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.748431 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.748439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:27.748621 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:27.749166 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:27.749184 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:27.751846 (+ 2662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":268,"threads_started":1}
W20260620 15:17:27.752275 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:27.744389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:60047 (local address 127.0.97.126:35025)
0620 15:17:27.744658 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:27.744664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:27.744841 (+ 177us) server_negotiation.cc:407] Connection header received
0620 15:17:27.744978 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:27.744982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:27.745052 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:27.745127 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:27.745819 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.746579 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.747771 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:27.748257 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:27.748380 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:27.752035 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:60047: BlockingRecv error: recv got EOF from 127.0.97.66:60047 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Jun 20 15:17:28 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:28 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:28.199905 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:28.188417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:59457)
0620 15:17:28.188948 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:28.188963 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:28.189072 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:28.189414 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:28.189417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:28.189809 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:28.190007 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:28.190019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.191091 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.191097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:28.192015 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:28.192026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.192717 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.192722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:28.192832 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:28.196457 (+ 3625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:28.196475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:28.199664 (+ 3189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":284,"threads_started":1}
W20260620 15:17:28.200042 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:28.188459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59457 (local address 127.0.97.126:35025)
0620 15:17:28.188672 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:28.188676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:28.189052 (+ 376us) server_negotiation.cc:407] Connection header received
0620 15:17:28.189223 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:28.189228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:28.189300 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:28.189377 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:28.190151 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.190946 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.192140 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.192591 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.192680 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:28.199839 (+ 7159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:59457: BlockingRecv error: recv got EOF from 127.0.97.67:59457 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 15:17:28 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:28 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:28.327812 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:28.319998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:58521)
0620 15:17:28.320534 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:28.320550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:28.320680 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:28.320969 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:28.320972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:28.321344 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:28.321491 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:28.321500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.322452 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.322456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:28.323357 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:28.323367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.324036 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.324040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:28.324168 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:28.324760 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:28.324777 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:28.327482 (+ 2705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":320,"threads_started":1}
W20260620 15:17:28.328024 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:28.320069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:58521 (local address 127.0.97.126:35025)
0620 15:17:28.320225 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:28.320229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:28.320653 (+ 424us) server_negotiation.cc:407] Connection header received
0620 15:17:28.320783 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:28.320787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:28.320844 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:28.320923 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:28.321616 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.322335 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.323480 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.323921 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.324012 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:28.327753 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:58521: BlockingRecv error: recv got EOF from 127.0.97.65:58521 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 15:17:28 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:28 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:28.761528 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:28.753197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:55381)
0620 15:17:28.753761 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:28.753776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:28.753880 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:28.754172 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:28.754198 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:28.754562 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:28.754708 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:28.754715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.755644 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.755648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:28.756595 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:28.756610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.757308 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.757313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:28.757434 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:28.757969 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:28.757987 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:28.761204 (+ 3217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":323,"threads_started":1}
W20260620 15:17:28.761595 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:28.753211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:55381 (local address 127.0.97.126:35025)
0620 15:17:28.753394 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:28.753399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:28.753843 (+ 444us) server_negotiation.cc:407] Connection header received
0620 15:17:28.753967 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:28.753971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:28.754028 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:28.754107 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:28.754820 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.755509 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.756748 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:28.757194 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:28.757283 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:28.761376 (+ 4093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:55381: BlockingRecv error: recv got EOF from 127.0.97.66:55381 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 15:17:29 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:29 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:29.209393 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:29.200846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:42913)
0620 15:17:29.201365 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:29.201380 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:29.201496 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:29.201809 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:29.201812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:29.202181 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:29.202326 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:29.202334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.203297 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.203301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:29.204237 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:29.204252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.205150 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.205156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:29.205302 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:29.206040 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:29.206057 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:29.209054 (+ 2997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":317,"threads_started":1}
W20260620 15:17:29.209471 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:29.200875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:42913 (local address 127.0.97.126:35025)
0620 15:17:29.201066 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:29.201070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:29.201451 (+ 381us) server_negotiation.cc:407] Connection header received
0620 15:17:29.201595 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:29.201598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:29.201686 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:29.201766 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:29.202447 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.203170 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.204369 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.205004 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.205117 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:29.209243 (+ 4126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:42913: BlockingRecv error: recv got EOF from 127.0.97.67:42913 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 15:17:29 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:29 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:29.337018 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:29.328768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:36233)
0620 15:17:29.329277 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:29.329292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:29.329398 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:29.329675 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:29.329679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:29.330035 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:29.330181 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:29.330189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.331130 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.331134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:29.332178 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:29.332191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.333213 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.333220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:29.333391 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:29.334029 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:29.334069 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:29.336789 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":260,"threads_started":1}
W20260620 15:17:29.337160 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:29.328863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:36233 (local address 127.0.97.126:35025)
0620 15:17:29.329047 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:29.329051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:29.329364 (+ 313us) server_negotiation.cc:407] Connection header received
0620 15:17:29.329493 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:29.329497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:29.329556 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:29.329635 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:29.330299 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.330994 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.332331 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.333005 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.333158 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:29.336944 (+ 3786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:36233: BlockingRecv error: recv got EOF from 127.0.97.65:36233 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 15:17:29 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:29 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:29.770054 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:29.762401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:57775)
0620 15:17:29.762856 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:29.762871 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:29.762974 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:29.763228 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:29.763232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:29.763591 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:29.763733 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:29.763740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.764758 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.764761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:29.765806 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:29.765818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.766541 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.766546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:29.766771 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:29.767314 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:29.767329 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:29.769817 (+ 2488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":245,"threads_started":1}
W20260620 15:17:29.770237 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:29.762606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:57775 (local address 127.0.97.126:35025)
0620 15:17:29.762794 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:29.762801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:29.762937 (+ 136us) server_negotiation.cc:407] Connection header received
0620 15:17:29.763052 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:29.763055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:29.763112 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:29.763183 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:29.763840 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.764636 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.765947 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:29.766376 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:29.766529 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:29.769993 (+ 3464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:57775: BlockingRecv error: recv got EOF from 127.0.97.66:57775 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 15:17:30 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:30 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:30.218519 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:30.210286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:47011)
0620 15:17:30.210824 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:30.210840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:30.210945 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:30.211226 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:30.211229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:30.211595 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:30.211745 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:30.211753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.212884 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.212888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:30.213811 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:30.213822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.214514 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.214519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:30.214644 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:30.215327 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:30.215345 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:30.218182 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":296,"threads_started":1}
W20260620 15:17:30.218644 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:30.210406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:47011 (local address 127.0.97.126:35025)
0620 15:17:30.210585 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:30.210589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:30.210910 (+ 321us) server_negotiation.cc:407] Connection header received
0620 15:17:30.211036 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:30.211039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:30.211103 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:30.211182 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:30.211899 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.212732 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.213952 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.214359 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.214453 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:30.218425 (+ 3972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:47011: BlockingRecv error: recv got EOF from 127.0.97.67:47011 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 15:17:30 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:30 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:30.346369 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:30.337990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:56333)
0620 15:17:30.338515 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:30.338533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:30.338693 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:30.339090 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:30.339098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:30.339498 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:30.339676 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:30.339686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.340821 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.340828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:30.341770 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:30.341783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.342516 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.342522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:30.342712 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:30.343306 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:30.343324 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:30.346063 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":264,"threads_started":1}
W20260620 15:17:30.346510 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:30.338122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56333 (local address 127.0.97.126:35025)
0620 15:17:30.338438 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:30.338445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:30.338619 (+ 174us) server_negotiation.cc:407] Connection header received
0620 15:17:30.338829 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:30.338835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:30.338907 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:30.339045 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:30.339845 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.340674 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.341923 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.342362 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.342517 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:30.346214 (+ 3697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:56333: BlockingRecv error: recv got EOF from 127.0.97.65:56333 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Jun 20 15:17:30 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:30 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:30.779290 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:30.770900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:40415)
0620 15:17:30.771423 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:30.771439 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:30.771592 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:30.771919 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:30.771923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:30.772283 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:30.772446 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:30.772454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.773464 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.773470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:30.774372 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:30.774382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.775063 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.775068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:30.775251 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:30.775827 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:30.775889 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:30.779076 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":272,"threads_started":1}
W20260620 15:17:30.779572 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:30.771043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:40415 (local address 127.0.97.126:35025)
0620 15:17:30.771277 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:30.771285 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:30.771528 (+ 243us) server_negotiation.cc:407] Connection header received
0620 15:17:30.771716 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:30.771722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:30.771787 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:30.771891 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:30.772617 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.773334 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.774518 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:30.774916 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:30.775048 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:30.779244 (+ 4196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:40415: BlockingRecv error: recv got EOF from 127.0.97.66:40415 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 15:17:31 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:31 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:31.228765 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:31.219363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.67:59583)
0620 15:17:31.219889 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:31.219907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:31.220022 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:31.220376 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:31.220380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:31.220784 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:31.220978 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:31.220986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.222544 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.222548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:31.223498 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:31.223509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.224593 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.224597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:31.224721 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:31.225373 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:31.225390 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:31.228441 (+ 3051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":295,"threads_started":1}
W20260620 15:17:31.228895 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:31.219469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59583 (local address 127.0.97.126:35025)
0620 15:17:31.219678 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:31.219687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:31.219992 (+ 305us) server_negotiation.cc:407] Connection header received
0620 15:17:31.220145 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:31.220152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:31.220237 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:31.220349 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:31.221142 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.222354 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.223651 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.224382 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.224549 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:31.228707 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:59583: BlockingRecv error: recv got EOF from 127.0.97.67:59583 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 15:17:31 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:31 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:31.355659 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:31.347666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.65:60921)
0620 15:17:31.348188 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:31.348204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:31.348304 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:31.348636 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:31.348639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:31.349011 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:31.349224 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:31.349233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.350219 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.350223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:31.351127 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:31.351138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.351772 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.351776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:31.351888 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:31.352434 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:31.352451 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:31.355269 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":272,"threads_started":1}
W20260620 15:17:31.355783 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:31.347707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:60921 (local address 127.0.97.126:35025)
0620 15:17:31.347875 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:31.347879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:31.348276 (+ 397us) server_negotiation.cc:407] Connection header received
0620 15:17:31.348398 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:31.348401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:31.348458 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:31.348602 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:31.349359 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.350093 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.351254 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.351654 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.351742 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:31.355506 (+ 3764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:60921: BlockingRecv error: recv got EOF from 127.0.97.65:60921 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 15:17:31 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:17:31 dist-test-slave-7s64 krb5kdc[8613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:17:31.787842 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:31.780342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35025 (local address 127.0.97.66:33153)
0620 15:17:31.780856 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:31.780873 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:31.780974 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:31.781255 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:31.781258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:31.781641 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:31.781824 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:31.781831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.782796 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.782799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:31.783710 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:31.783721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.784345 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.784349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:31.784454 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:31.785067 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:31.785084 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:31.787509 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:35025: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":288,"threads_started":1}
W20260620 15:17:31.787947 9349 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:31.780416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:33153 (local address 127.0.97.126:35025)
0620 15:17:31.780621 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:31.780625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:31.780943 (+ 318us) server_negotiation.cc:407] Connection header received
0620 15:17:31.781064 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:31.781067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:31.781133 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:31.781208 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:31.781935 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.782662 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.783820 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:31.784223 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:31.784308 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:31.787695 (+ 3387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:33153: BlockingRecv error: recv got EOF from 127.0.97.66:33153 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 15:17:32.066087 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 9200
I20260620 15:17:32.077543 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:32.067120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.0.97.65:39537)
0620 15:17:32.067636 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:32.067645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:32.067680 (+ 35us) server_negotiation.cc:407] Connection header received
0620 15:17:32.067771 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:32.067777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:32.067997 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:32.068149 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:32.069466 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:32.070521 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:32.071689 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:32.072243 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:32.072338 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:32.074590 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:32.074598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:32.074611 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:32.074647 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:32.077104 (+ 2457us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:17:32.077332 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53136: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":267,"threads_started":1}
W20260620 15:17:32.077666 9661 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53136: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:17:32.077762 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 9200: Not authorized: Client connection negotiation failed: client connection to 127.0.97.65:39537: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:17:32.077944 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 9200
I20260620 15:17:32.094416 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 9351
I20260620 15:17:32.105512 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:32.095238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.0.97.66:44519)
0620 15:17:32.095783 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:32.095790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:32.095821 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:17:32.095874 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:32.095879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:32.096091 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:32.096237 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:32.097422 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:32.098436 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:32.099573 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:32.100106 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:32.100200 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:32.102480 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:32.102489 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:32.102500 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:32.102536 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:32.105060 (+ 2524us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:17:32.105304 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36184: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
W20260620 15:17:32.105621 9662 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36184: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:17:32.105621 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 9351: Not authorized: Client connection negotiation failed: client connection to 127.0.97.66:44519: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:17:32.105875 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 9351
I20260620 15:17:32.122217 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 9483
I20260620 15:17:32.132876 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:32.122932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.0.97.67:45083)
0620 15:17:32.123496 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:32.123502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:32.123526 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:17:32.123585 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:32.123591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:32.123785 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:32.123924 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:32.125040 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:32.125999 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:32.127098 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:32.127613 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:32.127705 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:32.129918 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:32.129926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:32.129938 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:32.129966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:32.132236 (+ 2270us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:17:32.132457 (+ 221us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41648: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":294,"threads_started":1}
W20260620 15:17:32.132867 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 9483: Not authorized: Client connection negotiation failed: client connection to 127.0.97.67:45083: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:17:32.132963 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 9483
W20260620 15:17:32.132987 9663 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41648: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:17:32.148602 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 9131
I20260620 15:17:32.149688 9193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:32.260968 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 9131
2026-06-20T15:17:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20827 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T15:17:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:17:32Z Disabled control of system clock
I20260620 15:17:32.332851 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40781
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:33043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:40781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 15:17:32.637261 9668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:32.637605 9668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:32.637749 9668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:32.644771 9668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:17:32.644858 9668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:32.644922 9668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:32.645025 9668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:17:32.645089 9668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:17:32.657651 9668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:40781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:32.658854 9668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:32.660614 9668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:32.671949 9674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:32.672065 9673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:32.672571 9676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:32.672724 9668 server_base.cc:1034] running on GCE node
I20260620 15:17:32.673475 9668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:32.674664 9668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:32.675865 9668 hybrid_clock.cc:648] HybridClock initialized: now 1781968652675816 us; error 41 us; skew 500 ppm
I20260620 15:17:32.676148 9668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:17:32.678218 9668 webserver.cc:460] Webserver started at http://127.0.97.126:46139/ using document root <none> and password file <none>
I20260620 15:17:32.678807 9668 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:32.678890 9668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:32.679132 9668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:32.681835 9668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "73f9f8f1d67f4d269aebd32f451ed3fd"
format_stamp: "Formatted at 2026-06-20 15:17:32 on dist-test-slave-7s64"
server_key: "44cc32cf5ebca30d13a4eca3e7a3068a"
server_key_iv: "38c4690be4145dcac726213de4f33635"
server_key_version: "encryptionkey@0"
I20260620 15:17:32.682859 9668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "73f9f8f1d67f4d269aebd32f451ed3fd"
format_stamp: "Formatted at 2026-06-20 15:17:32 on dist-test-slave-7s64"
server_key: "44cc32cf5ebca30d13a4eca3e7a3068a"
server_key_iv: "38c4690be4145dcac726213de4f33635"
server_key_version: "encryptionkey@0"
I20260620 15:17:32.688191 9668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:17:32.691282 9682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:32.692603 9668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 15:17:32.692775 9668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "73f9f8f1d67f4d269aebd32f451ed3fd"
format_stamp: "Formatted at 2026-06-20 15:17:32 on dist-test-slave-7s64"
server_key: "44cc32cf5ebca30d13a4eca3e7a3068a"
server_key_iv: "38c4690be4145dcac726213de4f33635"
server_key_version: "encryptionkey@0"
I20260620 15:17:32.692974 9668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:32.703462 9668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:32.704178 9668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:32.704453 9668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:32.721908 9668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:40781
I20260620 15:17:32.721941 9733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:40781 every 8 connection(s)
I20260620 15:17:32.723106 9668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:17:32.726468 9734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:32.729133 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 9668
I20260620 15:17:32.729339 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:17:32.729928 389 external_mini_cluster.cc:1428] Setting key 6ee618e574968927398ec689cd892ca0
I20260620 15:17:32.737305 9734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd: Bootstrap starting.
I20260620 15:17:32.740358 9734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:32.741364 9734 log.cc:824] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:32.742834 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:32.731787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.0.97.126:40781)
0620 15:17:32.732666 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:32.732676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:32.732712 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:17:32.734175 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:32.734195 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:32.734611 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:32.734967 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:32.735532 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:32.736761 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:32.738153 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:32.738801 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:32.739016 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:32.740293 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:32.740301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:32.740313 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:32.740939 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:32.741007 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:32.741032 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:32.741159 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:32.741357 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 15:17:32.741634 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":271,"threads_started":1}
I20260620 15:17:32.744335 9734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd: No bootstrap required, opened a new log
I20260620 15:17:32.751730 9734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f9f8f1d67f4d269aebd32f451ed3fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40781 } }
I20260620 15:17:32.752007 9734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:32.752081 9734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73f9f8f1d67f4d269aebd32f451ed3fd, State: Initialized, Role: FOLLOWER
I20260620 15:17:32.752626 9734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f9f8f1d67f4d269aebd32f451ed3fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40781 } }
I20260620 15:17:32.752807 9734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:17:32.752897 9734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:17:32.753007 9734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:32.754944 9734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f9f8f1d67f4d269aebd32f451ed3fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40781 } }
I20260620 15:17:32.755419 9734 leader_election.cc:302] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73f9f8f1d67f4d269aebd32f451ed3fd; no voters:
I20260620 15:17:32.756080 9734 leader_election.cc:288] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:17:32.756173 9739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:32.756860 9739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [term 1 LEADER]: Becoming Leader. State: Replica: 73f9f8f1d67f4d269aebd32f451ed3fd, State: Running, Role: LEADER
I20260620 15:17:32.757408 9739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f9f8f1d67f4d269aebd32f451ed3fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40781 } }
I20260620 15:17:32.758217 9734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:17:32.760212 9740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73f9f8f1d67f4d269aebd32f451ed3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f9f8f1d67f4d269aebd32f451ed3fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40781 } } }
I20260620 15:17:32.760603 9740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:32.760303 9741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73f9f8f1d67f4d269aebd32f451ed3fd. Latest consensus state: current_term: 1 leader_uuid: "73f9f8f1d67f4d269aebd32f451ed3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73f9f8f1d67f4d269aebd32f451ed3fd" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40781 } } }
I20260620 15:17:32.760710 9741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:32.761782 9745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:17:32.767639 9745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:17:32.776872 9745 catalog_manager.cc:1347] Generated new cluster ID: 3c0284d94b494549b9f264cdb55019e8
I20260620 15:17:32.776975 9745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:17:32.791465 9745 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:17:32.793107 9745 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:17:32.801276 9745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73f9f8f1d67f4d269aebd32f451ed3fd: Generated new TSK 0
I20260620 15:17:32.802135 9745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:17:32.811064 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40781
--builtin_ntp_servers=127.0.97.84:33043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 15:17:33.126519 9758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:33.126768 9758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:33.126847 9758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:33.133852 9758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:33.133980 9758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:33.134162 9758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:17:33.146085 9758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:33.147260 9758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:33.148991 9758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:33.161458 9764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:33.161669 9763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:33.161490 9766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:33.162321 9758 server_base.cc:1034] running on GCE node
I20260620 15:17:33.162829 9758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:33.163635 9758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:33.164825 9758 hybrid_clock.cc:648] HybridClock initialized: now 1781968653164793 us; error 32 us; skew 500 ppm
I20260620 15:17:33.165074 9758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:17:33.167030 9758 webserver.cc:460] Webserver started at http://127.0.97.65:33689/ using document root <none> and password file <none>
I20260620 15:17:33.167636 9758 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:33.167764 9758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:33.168053 9758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:33.170778 9758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "e4e622ee0ee343a69257b8cef5bbeec9"
format_stamp: "Formatted at 2026-06-20 15:17:33 on dist-test-slave-7s64"
server_key: "6ab28b344cfb04b1a51f7c2ad5a56b61"
server_key_iv: "0e2c3aa216157837dbbe20c6b8028ea2"
server_key_version: "encryptionkey@0"
I20260620 15:17:33.171471 9758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "e4e622ee0ee343a69257b8cef5bbeec9"
format_stamp: "Formatted at 2026-06-20 15:17:33 on dist-test-slave-7s64"
server_key: "6ab28b344cfb04b1a51f7c2ad5a56b61"
server_key_iv: "0e2c3aa216157837dbbe20c6b8028ea2"
server_key_version: "encryptionkey@0"
I20260620 15:17:33.176582 9758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 15:17:33.179651 9772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:33.180946 9758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:17:33.181069 9758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "e4e622ee0ee343a69257b8cef5bbeec9"
format_stamp: "Formatted at 2026-06-20 15:17:33 on dist-test-slave-7s64"
server_key: "6ab28b344cfb04b1a51f7c2ad5a56b61"
server_key_iv: "0e2c3aa216157837dbbe20c6b8028ea2"
server_key_version: "encryptionkey@0"
I20260620 15:17:33.181232 9758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:33.213044 9758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:33.213763 9758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:33.213968 9758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:33.214762 9758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:33.216056 9758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:33.216138 9758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:33.216213 9758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:33.216266 9758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:33.248190 9758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:39421
I20260620 15:17:33.248284 9884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:39421 every 8 connection(s)
I20260620 15:17:33.249398 9758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:17:33.250481 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 9758
I20260620 15:17:33.250684 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:17:33.251348 389 external_mini_cluster.cc:1428] Setting key 4098a11e66d12e9b8f355600ff8f414b
I20260620 15:17:33.254266 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40781
--builtin_ntp_servers=127.0.97.84:33043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 15:17:33.262010 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:33.251962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:43769 (local address 127.0.97.126:40781)
0620 15:17:33.252707 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:33.252713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:33.253830 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 15:17:33.255603 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:33.255608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:33.255686 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:33.255811 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:33.256878 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:33.257604 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:33.259274 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:33.259696 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:33.259841 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:33.261190 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:33.261195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:33.261198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:33.261214 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:33.261252 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:33.261264 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:33.261372 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:33.261611 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 15:17:33.261699 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":527,"threads_started":1}
I20260620 15:17:33.262928 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:33.252683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40781 (local address 127.0.97.65:43769)
0620 15:17:33.253686 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:33.253725 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:33.255344 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:33.256031 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:33.256039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:33.256128 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:33.256720 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:33.256737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:33.257758 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:33.257762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:33.259127 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:33.259139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:33.259917 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:33.259925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:33.260177 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:33.260866 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:17:33.260872 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:33.260921 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:17:33.261014 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:33.261371 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:33.261383 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:33.261394 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:17:33.261584 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 15:17:33.261932 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":319,"threads_started":1}
I20260620 15:17:33.264533 9885 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40781
I20260620 15:17:33.264815 9885 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:33.265491 9885 heartbeater.cc:511] Master 127.0.97.126:40781 requested a full tablet report, sending...
I20260620 15:17:33.267511 9699 ts_manager.cc:194] Registered new tserver with Master: e4e622ee0ee343a69257b8cef5bbeec9 (127.0.97.65:39421)
I20260620 15:17:33.269771 9699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:43769
W20260620 15:17:33.566377 9889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:33.566620 9889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:33.566700 9889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:33.573633 9889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:33.573720 9889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:33.573814 9889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:17:33.582237 9889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:33.583370 9889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:33.585225 9889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:33.600693 9894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:33.601754 9895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:33.601814 9897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:33.602591 9889 server_base.cc:1034] running on GCE node
I20260620 15:17:33.603197 9889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:33.604087 9889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:33.605305 9889 hybrid_clock.cc:648] HybridClock initialized: now 1781968653605250 us; error 48 us; skew 500 ppm
I20260620 15:17:33.605613 9889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:17:33.607654 9889 webserver.cc:460] Webserver started at http://127.0.97.66:37997/ using document root <none> and password file <none>
I20260620 15:17:33.608309 9889 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:33.608438 9889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:33.608777 9889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:33.611475 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "a1c814446a6444438183ad9f4f76f37d"
format_stamp: "Formatted at 2026-06-20 15:17:33 on dist-test-slave-7s64"
server_key: "c4630c1633f0794ddd9d8d6476b63894"
server_key_iv: "8adc7ddf6ce7d22881bdfa736f067941"
server_key_version: "encryptionkey@0"
I20260620 15:17:33.612232 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "a1c814446a6444438183ad9f4f76f37d"
format_stamp: "Formatted at 2026-06-20 15:17:33 on dist-test-slave-7s64"
server_key: "c4630c1633f0794ddd9d8d6476b63894"
server_key_iv: "8adc7ddf6ce7d22881bdfa736f067941"
server_key_version: "encryptionkey@0"
I20260620 15:17:33.616968 9889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:17:33.620028 9903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:33.621248 9889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 15:17:33.621413 9889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "a1c814446a6444438183ad9f4f76f37d"
format_stamp: "Formatted at 2026-06-20 15:17:33 on dist-test-slave-7s64"
server_key: "c4630c1633f0794ddd9d8d6476b63894"
server_key_iv: "8adc7ddf6ce7d22881bdfa736f067941"
server_key_version: "encryptionkey@0"
I20260620 15:17:33.621620 9889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:33.637323 9889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:33.638041 9889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:33.638307 9889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:33.639130 9889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:33.640471 9889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:33.640622 9889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:33.640750 9889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:33.640838 9889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:33.674381 9889 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:34647
I20260620 15:17:33.674470 10015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:34647 every 8 connection(s)
I20260620 15:17:33.675709 9889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:17:33.684844 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 9889
I20260620 15:17:33.685043 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:17:33.685662 389 external_mini_cluster.cc:1428] Setting key ee49263c19da5367f7b7a74e5c9c12be
I20260620 15:17:33.689265 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:33.678176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:55727 (local address 127.0.97.126:40781)
0620 15:17:33.678344 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:33.678348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:33.679781 (+ 1433us) server_negotiation.cc:407] Connection header received
0620 15:17:33.681548 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:33.681554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:33.681652 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:33.681768 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:33.682771 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:33.683482 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:33.685474 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:33.686242 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:33.686391 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:33.688305 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:33.688312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:33.688316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:33.688337 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:33.688370 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:33.688388 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:33.688451 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:33.688800 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 15:17:33.688893 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:17:33.689281 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40781
--builtin_ntp_servers=127.0.97.84:33043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 15:17:33.691210 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:33.678631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40781 (local address 127.0.97.66:55727)
0620 15:17:33.679610 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:33.679645 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:33.681285 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:33.681983 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:33.681992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:33.682072 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:33.682599 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:33.682614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:33.683632 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:33.683644 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:33.685251 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:33.685275 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:33.686595 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:33.686606 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:33.687118 (+ 512us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:33.687947 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:17:33.687950 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:33.687984 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:17:33.688095 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:33.688564 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:33.688575 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:33.688582 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:17:33.689468 (+ 886us) client_negotiation.cc:240] Negotiation successful
0620 15:17:33.689862 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":316,"threads_started":1}
I20260620 15:17:33.692561 10016 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40781
I20260620 15:17:33.692838 10016 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:33.693539 10016 heartbeater.cc:511] Master 127.0.97.126:40781 requested a full tablet report, sending...
I20260620 15:17:33.694850 9699 ts_manager.cc:194] Registered new tserver with Master: a1c814446a6444438183ad9f4f76f37d (127.0.97.66:34647)
I20260620 15:17:33.695627 9699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:55727
W20260620 15:17:34.004274 10019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:34.004601 10019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:34.004747 10019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:34.011701 10019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:34.011826 10019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:34.011997 10019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:17:34.020628 10019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:34.021780 10019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:34.023480 10019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:34.036168 10024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:34.036134 10027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:34.036198 10025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:34.036892 10019 server_base.cc:1034] running on GCE node
I20260620 15:17:34.037447 10019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:34.038267 10019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:34.039476 10019 hybrid_clock.cc:648] HybridClock initialized: now 1781968654039434 us; error 37 us; skew 500 ppm
I20260620 15:17:34.039814 10019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:17:34.041903 10019 webserver.cc:460] Webserver started at http://127.0.97.67:35173/ using document root <none> and password file <none>
I20260620 15:17:34.042639 10019 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:34.042755 10019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:34.043062 10019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:34.045728 10019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "981b1537810f478ca868769a4d3b4383"
format_stamp: "Formatted at 2026-06-20 15:17:34 on dist-test-slave-7s64"
server_key: "bac7352c7d1cc05ea5fdab409331cef7"
server_key_iv: "4f19212513ed345d9cdc1dbb52e9b4df"
server_key_version: "encryptionkey@0"
I20260620 15:17:34.046444 10019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "981b1537810f478ca868769a4d3b4383"
format_stamp: "Formatted at 2026-06-20 15:17:34 on dist-test-slave-7s64"
server_key: "bac7352c7d1cc05ea5fdab409331cef7"
server_key_iv: "4f19212513ed345d9cdc1dbb52e9b4df"
server_key_version: "encryptionkey@0"
I20260620 15:17:34.051039 10019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 15:17:34.054123 10033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:34.055325 10019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:17:34.055482 10019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "981b1537810f478ca868769a4d3b4383"
format_stamp: "Formatted at 2026-06-20 15:17:34 on dist-test-slave-7s64"
server_key: "bac7352c7d1cc05ea5fdab409331cef7"
server_key_iv: "4f19212513ed345d9cdc1dbb52e9b4df"
server_key_version: "encryptionkey@0"
I20260620 15:17:34.055684 10019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:34.095387 10019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:34.096287 10019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:34.096578 10019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:34.097527 10019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:34.098944 10019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:34.099068 10019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:34.099198 10019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:34.099285 10019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:34.130869 10019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:38855
I20260620 15:17:34.130947 10145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:38855 every 8 connection(s)
I20260620 15:17:34.132059 10019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:17:34.140602 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 10019
I20260620 15:17:34.140785 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:17:34.141198 389 external_mini_cluster.cc:1428] Setting key 90ed1f065736ea748fd7816ab91be4dd
I20260620 15:17:34.144443 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:34.134564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:36103 (local address 127.0.97.126:40781)
0620 15:17:34.134740 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:34.134745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:34.136054 (+ 1309us) server_negotiation.cc:407] Connection header received
0620 15:17:34.137813 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:34.137819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:34.137906 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:34.138012 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:34.138986 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:34.139705 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:34.141418 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:34.141961 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:34.142114 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:34.143681 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:34.143686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:34.143690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:34.143705 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:34.143732 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:34.143743 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:34.143835 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:34.144038 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 15:17:34.144131 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:17:34.145341 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:34.135063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40781 (local address 127.0.97.67:36103)
0620 15:17:34.135909 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:34.135944 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:34.137555 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:34.138225 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:34.138233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:34.138316 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:34.138826 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:34.138840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:34.139865 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:34.139870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:34.141261 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:34.141276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:34.142272 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:34.142283 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:34.142594 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:34.143394 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:17:34.143397 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:34.143431 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:17:34.143499 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:34.143833 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:34.143844 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:34.143854 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:17:34.144030 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 15:17:34.144385 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":255,"threads_started":1}
I20260620 15:17:34.146627 10146 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40781
I20260620 15:17:34.146929 10146 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:34.147635 10146 heartbeater.cc:511] Master 127.0.97.126:40781 requested a full tablet report, sending...
I20260620 15:17:34.148919 9699 ts_manager.cc:194] Registered new tserver with Master: 981b1537810f478ca868769a4d3b4383 (127.0.97.67:38855)
I20260620 15:17:34.149679 9699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:36103
I20260620 15:17:34.156284 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:17:34.161952 9887 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:34.160719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.0.97.126:40781)
0620 15:17:34.160863 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:34.160867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:34.161260 (+ 393us) server_negotiation.cc:407] Connection header received
0620 15:17:34.161369 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:34.161372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:34.161423 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:34.161497 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:34.161785 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59932: BlockingRecv error: recv got EOF from 127.0.0.1:59932 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:17:34.162657 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 9758
I20260620 15:17:34.169819 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:34.163534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.0.97.65:39421)
0620 15:17:34.163996 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:34.164004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:34.164038 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:17:34.164085 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:34.164090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:34.164266 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:34.164380 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:34.164756 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:34.165688 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:34.166800 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:34.167316 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:34.167409 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:34.168412 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:34.168419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:34.168430 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:34.169045 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:34.169102 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:34.169121 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:34.169214 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:34.169373 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 15:17:34.169465 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":237,"threads_started":1}
I20260620 15:17:34.170861 9880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:34.250874 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 9758
I20260620 15:17:34.266762 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 9889
I20260620 15:17:34.274678 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:34.267591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.97.66:34647)
0620 15:17:34.268127 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:34.268143 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:34.268174 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:17:34.268239 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:34.268246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:34.268583 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:34.268750 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:34.269184 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:34.270188 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:34.271392 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:34.271939 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:34.272076 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:34.273208 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:34.273216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:34.273227 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:34.273785 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:34.273827 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:34.273845 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:34.273972 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:34.274222 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 15:17:34.274361 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":305,"threads_started":1}
I20260620 15:17:34.275981 10011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:34.356117 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 9889
I20260620 15:17:34.371994 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 10019
I20260620 15:17:34.379819 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:34.373107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.0.97.67:38855)
0620 15:17:34.373737 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:34.373744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:34.373773 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:17:34.373836 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:34.373841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:34.374079 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:34.374199 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:34.374573 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:34.375606 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:34.376861 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:34.377406 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:34.377504 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:34.378466 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:34.378473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:34.378486 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:34.379053 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:34.379095 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:34.379112 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:34.379199 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:34.379335 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 15:17:34.379459 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":351,"threads_started":1}
I20260620 15:17:34.381114 10141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:34.463174 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 10019
I20260620 15:17:34.479302 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 9668
I20260620 15:17:34.481022 9729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:34.581072 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 9668
2026-06-20T15:17:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2315 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T15:17:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:17:34Z Disabled control of system clock
I20260620 15:17:34.647871 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43055
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:36993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 15:17:34.955636 10165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:34.955902 10165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:34.955984 10165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:34.963125 10165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:17:34.963229 10165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:34.963266 10165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:34.963315 10165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:17:34.963357 10165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:17:34.972268 10165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43055
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:34.973655 10165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:34.975323 10165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:34.986469 10170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:34.986534 10171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:34.986478 10173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:34.987648 10165 server_base.cc:1034] running on GCE node
I20260620 15:17:34.988413 10165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:34.989786 10165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:34.991039 10165 hybrid_clock.cc:648] HybridClock initialized: now 1781968654990991 us; error 57 us; skew 500 ppm
I20260620 15:17:34.991430 10165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:17:34.993973 10165 webserver.cc:460] Webserver started at http://127.0.97.126:43071/ using document root <none> and password file <none>
I20260620 15:17:34.994704 10165 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:34.994837 10165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:34.995193 10165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:34.998293 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "0b8097a313b8442698c7114a0860569e"
format_stamp: "Formatted at 2026-06-20 15:17:34 on dist-test-slave-7s64"
server_key: "62999eb030e3f9bb313c0862ddc9d94e"
server_key_iv: "3451168248bf69224b6d9b5c16dd328c"
server_key_version: "encryptionkey@0"
I20260620 15:17:34.999148 10165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "0b8097a313b8442698c7114a0860569e"
format_stamp: "Formatted at 2026-06-20 15:17:34 on dist-test-slave-7s64"
server_key: "62999eb030e3f9bb313c0862ddc9d94e"
server_key_iv: "3451168248bf69224b6d9b5c16dd328c"
server_key_version: "encryptionkey@0"
I20260620 15:17:35.004422 10165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:17:35.007999 10179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:35.009433 10165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:17:35.009649 10165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "0b8097a313b8442698c7114a0860569e"
format_stamp: "Formatted at 2026-06-20 15:17:34 on dist-test-slave-7s64"
server_key: "62999eb030e3f9bb313c0862ddc9d94e"
server_key_iv: "3451168248bf69224b6d9b5c16dd328c"
server_key_version: "encryptionkey@0"
I20260620 15:17:35.009891 10165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:35.015795 10165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:35.016714 10165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:35.017033 10165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:35.035946 10165 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43055
I20260620 15:17:35.035965 10230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43055 every 8 connection(s)
I20260620 15:17:35.037446 10165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:17:35.041355 10231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:35.044586 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 10165
I20260620 15:17:35.044788 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:17:35.045606 389 external_mini_cluster.cc:1428] Setting key 48b3b49a1ac9d3911b162248f7e3f364
I20260620 15:17:35.054087 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:35.047835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.0.97.126:43055)
0620 15:17:35.048892 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:35.048904 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:35.048946 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:17:35.050603 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:35.050627 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:35.051075 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:35.051431 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:35.052470 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:35.052476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:35.052545 (+ 69us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:35.052561 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:35.052623 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:35.052641 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:35.052819 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:35.052929 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 15:17:35.053200 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":403,"threads_started":1}
I20260620 15:17:35.054489 10231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e: Bootstrap starting.
I20260620 15:17:35.057397 10231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:35.058336 10231 log.cc:824] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:35.060974 10231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e: No bootstrap required, opened a new log
I20260620 15:17:35.067639 10231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8097a313b8442698c7114a0860569e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43055 } }
I20260620 15:17:35.067992 10231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:35.068090 10231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b8097a313b8442698c7114a0860569e, State: Initialized, Role: FOLLOWER
I20260620 15:17:35.068725 10231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8097a313b8442698c7114a0860569e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43055 } }
I20260620 15:17:35.068998 10231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:17:35.069131 10231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:17:35.069303 10231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:35.071282 10231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8097a313b8442698c7114a0860569e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43055 } }
I20260620 15:17:35.071758 10231 leader_election.cc:302] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b8097a313b8442698c7114a0860569e; no voters:
I20260620 15:17:35.072331 10231 leader_election.cc:288] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:17:35.072464 10236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:35.073205 10236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [term 1 LEADER]: Becoming Leader. State: Replica: 0b8097a313b8442698c7114a0860569e, State: Running, Role: LEADER
I20260620 15:17:35.073793 10236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8097a313b8442698c7114a0860569e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43055 } }
I20260620 15:17:35.074390 10231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:17:35.076258 10238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b8097a313b8442698c7114a0860569e. Latest consensus state: current_term: 1 leader_uuid: "0b8097a313b8442698c7114a0860569e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8097a313b8442698c7114a0860569e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43055 } } }
I20260620 15:17:35.076622 10238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:35.076929 10237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b8097a313b8442698c7114a0860569e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b8097a313b8442698c7114a0860569e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43055 } } }
I20260620 15:17:35.077488 10237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:35.085004 10241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:17:35.089150 10241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:17:35.097568 10241 catalog_manager.cc:1347] Generated new cluster ID: d81b315c85724ab78a1e3da83f207c1a
I20260620 15:17:35.097671 10241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:17:35.122457 10241 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:17:35.122864 10241 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:17:35.130925 10241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b8097a313b8442698c7114a0860569e: Generated new TSK 0
I20260620 15:17:35.131711 10241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:17:35.143006 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43055
--builtin_ntp_servers=127.0.97.84:36993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 15:17:35.458676 10256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:35.458920 10256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:35.459075 10256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:35.465993 10256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:35.466107 10256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:35.466259 10256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:17:35.474809 10256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:35.475922 10256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:35.477568 10256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:35.490087 10261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:35.490000 10264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:35.490125 10262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:35.490900 10256 server_base.cc:1034] running on GCE node
I20260620 15:17:35.491472 10256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:35.492280 10256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:35.493480 10256 hybrid_clock.cc:648] HybridClock initialized: now 1781968655493426 us; error 64 us; skew 500 ppm
I20260620 15:17:35.493772 10256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:17:35.495836 10256 webserver.cc:460] Webserver started at http://127.0.97.65:36545/ using document root <none> and password file <none>
I20260620 15:17:35.496451 10256 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:35.496614 10256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:35.496905 10256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:35.499578 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "d2a06a77871f4e6fa60b7aa1ea0bf779"
format_stamp: "Formatted at 2026-06-20 15:17:35 on dist-test-slave-7s64"
server_key: "829f7a4c5da27f197e22d5031caecfc3"
server_key_iv: "15b69e183eda7965b06cc9fa011ee908"
server_key_version: "encryptionkey@0"
I20260620 15:17:35.500312 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "d2a06a77871f4e6fa60b7aa1ea0bf779"
format_stamp: "Formatted at 2026-06-20 15:17:35 on dist-test-slave-7s64"
server_key: "829f7a4c5da27f197e22d5031caecfc3"
server_key_iv: "15b69e183eda7965b06cc9fa011ee908"
server_key_version: "encryptionkey@0"
I20260620 15:17:35.504894 10256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:17:35.507965 10270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:35.509270 10256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:17:35.509426 10256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "d2a06a77871f4e6fa60b7aa1ea0bf779"
format_stamp: "Formatted at 2026-06-20 15:17:35 on dist-test-slave-7s64"
server_key: "829f7a4c5da27f197e22d5031caecfc3"
server_key_iv: "15b69e183eda7965b06cc9fa011ee908"
server_key_version: "encryptionkey@0"
I20260620 15:17:35.509637 10256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:35.514871 10256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:35.515563 10256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:35.515765 10256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:35.516597 10256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:35.517782 10256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:35.517863 10256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:35.517928 10256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:35.517980 10256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:35.549252 10256 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:35059
I20260620 15:17:35.549312 10382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:35059 every 8 connection(s)
I20260620 15:17:35.550436 10256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:17:35.555085 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 10256
I20260620 15:17:35.555261 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:17:35.556008 389 external_mini_cluster.cc:1428] Setting key a8b55066778855335408ff293684e5e9
I20260620 15:17:35.558359 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:35.552806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:50957 (local address 127.0.97.126:43055)
0620 15:17:35.552967 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:35.552972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:35.554177 (+ 1205us) server_negotiation.cc:407] Connection header received
0620 15:17:35.556355 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:35.556360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:35.556411 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:35.556543 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:35.557715 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:35.557720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:35.557725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:35.557742 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:35.557778 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:35.557793 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:35.557901 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:35.558122 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 15:17:35.558168 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:17:35.558847 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43055
--builtin_ntp_servers=127.0.97.84:36993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 15:17:35.559549 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:35.553240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43055 (local address 127.0.97.65:50957)
0620 15:17:35.554035 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:35.554068 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:35.556068 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:35.556743 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:35.556751 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:35.556832 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:35.557471 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:17:35.557473 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:35.557506 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:17:35.557589 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:35.557895 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:35.557904 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:35.557912 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:17:35.558176 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 15:17:35.558404 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":230,"threads_started":1}
I20260620 15:17:35.560902 10383 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43055
I20260620 15:17:35.561177 10383 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:35.561685 10383 heartbeater.cc:511] Master 127.0.97.126:43055 requested a full tablet report, sending...
I20260620 15:17:35.563484 10196 ts_manager.cc:194] Registered new tserver with Master: d2a06a77871f4e6fa60b7aa1ea0bf779 (127.0.97.65:35059)
W20260620 15:17:35.869086 10386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:35.869365 10386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:35.869499 10386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:35.876727 10386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:35.876813 10386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:35.876991 10386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:17:35.886242 10386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:35.887421 10386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:35.889168 10386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:35.901645 10391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:35.901710 10392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:35.901683 10394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:35.902431 10386 server_base.cc:1034] running on GCE node
I20260620 15:17:35.902959 10386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:35.903790 10386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:35.905004 10386 hybrid_clock.cc:648] HybridClock initialized: now 1781968655904963 us; error 51 us; skew 500 ppm
I20260620 15:17:35.905280 10386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:17:35.907457 10386 webserver.cc:460] Webserver started at http://127.0.97.66:33573/ using document root <none> and password file <none>
I20260620 15:17:35.908092 10386 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:35.908209 10386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:35.908536 10386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:35.912595 10386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "72a8c3466cbd4cbfa97cc55c556e079f"
format_stamp: "Formatted at 2026-06-20 15:17:35 on dist-test-slave-7s64"
server_key: "b15d2a84d3b1de828eb60a0fc3623f49"
server_key_iv: "e51c4a2528961a2dbc85766f8795ea0e"
server_key_version: "encryptionkey@0"
I20260620 15:17:35.913424 10386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "72a8c3466cbd4cbfa97cc55c556e079f"
format_stamp: "Formatted at 2026-06-20 15:17:35 on dist-test-slave-7s64"
server_key: "b15d2a84d3b1de828eb60a0fc3623f49"
server_key_iv: "e51c4a2528961a2dbc85766f8795ea0e"
server_key_version: "encryptionkey@0"
I20260620 15:17:35.918154 10386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 15:17:35.921278 10400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:35.922467 10386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:17:35.922634 10386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "72a8c3466cbd4cbfa97cc55c556e079f"
format_stamp: "Formatted at 2026-06-20 15:17:35 on dist-test-slave-7s64"
server_key: "b15d2a84d3b1de828eb60a0fc3623f49"
server_key_iv: "e51c4a2528961a2dbc85766f8795ea0e"
server_key_version: "encryptionkey@0"
I20260620 15:17:35.922839 10386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:35.928215 10386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:35.928965 10386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:35.929216 10386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:35.930019 10386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:35.931238 10386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:35.931353 10386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:35.931460 10386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:35.931543 10386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:35.963449 10386 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37603
I20260620 15:17:35.963529 10512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37603 every 8 connection(s)
I20260620 15:17:35.964663 10386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:17:35.968137 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 10386
I20260620 15:17:35.968297 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:17:35.968875 389 external_mini_cluster.cc:1428] Setting key 9b7700aef99bf4a8a49c2025e9481563
I20260620 15:17:35.971690 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43055
--builtin_ntp_servers=127.0.97.84:36993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 15:17:35.972952 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:35.967045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:40727 (local address 127.0.97.126:43055)
0620 15:17:35.967206 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:35.967211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:35.968778 (+ 1567us) server_negotiation.cc:407] Connection header received
0620 15:17:35.970740 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:35.970746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:35.970807 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:35.970914 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:35.972217 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:35.972223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:35.972227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:35.972244 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:35.972279 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:35.972291 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:35.972397 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:35.972621 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 15:17:35.972685 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:17:35.973819 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:35.967458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43055 (local address 127.0.97.66:40727)
0620 15:17:35.968578 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:35.968624 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:35.970466 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:35.971128 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:35.971137 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:35.971221 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:35.971926 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:17:35.971932 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:35.971982 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:17:35.972068 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:35.972385 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:35.972395 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:35.972402 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:17:35.972700 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 15:17:35.972934 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":287,"threads_started":1}
I20260620 15:17:35.975078 10513 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43055
I20260620 15:17:35.975363 10513 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:35.975898 10513 heartbeater.cc:511] Master 127.0.97.126:43055 requested a full tablet report, sending...
I20260620 15:17:35.977257 10196 ts_manager.cc:194] Registered new tserver with Master: 72a8c3466cbd4cbfa97cc55c556e079f (127.0.97.66:37603)
W20260620 15:17:36.291713 10516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:36.291949 10516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:36.292095 10516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:36.298888 10516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:36.298966 10516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:36.299144 10516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:17:36.307559 10516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:36.308682 10516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:36.310245 10516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:36.322491 10521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:36.322541 10522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:36.322579 10524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:36.323351 10516 server_base.cc:1034] running on GCE node
I20260620 15:17:36.323822 10516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:36.324633 10516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:36.325824 10516 hybrid_clock.cc:648] HybridClock initialized: now 1781968656325782 us; error 40 us; skew 500 ppm
I20260620 15:17:36.326110 10516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:17:36.328212 10516 webserver.cc:460] Webserver started at http://127.0.97.67:37279/ using document root <none> and password file <none>
I20260620 15:17:36.328881 10516 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:36.328981 10516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:36.329272 10516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:36.331781 10516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "8ae9b103e47843b7985cdbd43f38401e"
format_stamp: "Formatted at 2026-06-20 15:17:36 on dist-test-slave-7s64"
server_key: "e6aaa31c7cce41ea957f8a3099ed69bb"
server_key_iv: "cef3200f559f489b0395dbc83f9deba9"
server_key_version: "encryptionkey@0"
I20260620 15:17:36.332548 10516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "8ae9b103e47843b7985cdbd43f38401e"
format_stamp: "Formatted at 2026-06-20 15:17:36 on dist-test-slave-7s64"
server_key: "e6aaa31c7cce41ea957f8a3099ed69bb"
server_key_iv: "cef3200f559f489b0395dbc83f9deba9"
server_key_version: "encryptionkey@0"
I20260620 15:17:36.337096 10516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 15:17:36.340227 10530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:36.341461 10516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:17:36.341629 10516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "8ae9b103e47843b7985cdbd43f38401e"
format_stamp: "Formatted at 2026-06-20 15:17:36 on dist-test-slave-7s64"
server_key: "e6aaa31c7cce41ea957f8a3099ed69bb"
server_key_iv: "cef3200f559f489b0395dbc83f9deba9"
server_key_version: "encryptionkey@0"
I20260620 15:17:36.341830 10516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:36.347009 10516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:36.347793 10516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:36.348039 10516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:36.348912 10516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:36.350131 10516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:36.350245 10516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:36.350353 10516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:36.350431 10516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:36.380194 10516 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39039
I20260620 15:17:36.380254 10642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39039 every 8 connection(s)
I20260620 15:17:36.381405 10516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:17:36.382244 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 10516
I20260620 15:17:36.382403 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:17:36.382913 389 external_mini_cluster.cc:1428] Setting key cc80893656e46bc0bf55a01ab3c74391
I20260620 15:17:36.389848 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:36.383921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:46209 (local address 127.0.97.126:43055)
0620 15:17:36.384149 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:36.384156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:36.385540 (+ 1384us) server_negotiation.cc:407] Connection header received
0620 15:17:36.387245 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:36.387251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:36.387313 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:36.387417 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:36.388996 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:36.389003 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:36.389009 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:36.389029 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:36.389093 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:36.389111 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:36.389275 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:36.389419 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 15:17:36.389501 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:17:36.390664 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:36.384558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43055 (local address 127.0.97.67:46209)
0620 15:17:36.385401 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:36.385437 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:36.387007 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:36.387651 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:36.387662 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:36.387762 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:36.388640 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:17:36.388643 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:36.388692 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:17:36.388786 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:36.389222 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:36.389231 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:36.389238 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:17:36.389487 (+ 249us) client_negotiation.cc:240] Negotiation successful
0620 15:17:36.389703 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":284,"threads_started":1}
I20260620 15:17:36.391762 10643 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43055
I20260620 15:17:36.391989 10643 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:36.392449 10643 heartbeater.cc:511] Master 127.0.97.126:43055 requested a full tablet report, sending...
I20260620 15:17:36.393630 10196 ts_manager.cc:194] Registered new tserver with Master: 8ae9b103e47843b7985cdbd43f38401e (127.0.97.67:39039)
I20260620 15:17:36.397995 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:17:36.403573 10234 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:17:36.402241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.0.97.126:43055)
0620 15:17:36.402356 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:36.402360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:36.402870 (+ 510us) server_negotiation.cc:407] Connection header received
0620 15:17:36.402980 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:36.402983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:36.403047 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:36.403116 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:36.403356 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60110: BlockingRecv error: recv got EOF from 127.0.0.1:60110 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 15:17:36.404131 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 10256
I20260620 15:17:36.407184 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:36.405013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.0.97.65:35059)
0620 15:17:36.405486 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:36.405491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:36.405514 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:17:36.405565 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:36.405572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:36.405728 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:36.405837 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:36.406626 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:36.406632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:36.406640 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:36.406657 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:36.406692 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:36.406706 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:36.406799 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:36.406877 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 15:17:36.406942 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260620 15:17:36.408262 10378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:36.487538 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 10256
I20260620 15:17:36.503073 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 10386
I20260620 15:17:36.506557 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:36.503860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.0.97.66:37603)
0620 15:17:36.504375 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:36.504382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:36.504409 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:17:36.504469 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:36.504475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:36.504758 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:36.504892 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:36.505872 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:36.505880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:36.505893 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:36.505917 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:36.505965 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:36.505981 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:36.506092 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:36.506246 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 15:17:36.506328 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":260,"threads_started":1}
I20260620 15:17:36.507850 10508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:36.587090 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 10386
I20260620 15:17:36.602735 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 10516
I20260620 15:17:36.606725 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:36.603709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.0.97.67:39039)
0620 15:17:36.604351 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:36.604358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:36.604386 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:17:36.604444 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:36.604450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:36.604736 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:36.604898 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:36.606007 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:36.606013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:36.606024 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:17:36.606038 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:36.606075 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:17:36.606088 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:36.606274 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:36.606343 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 15:17:36.606437 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":328,"threads_started":1}
I20260620 15:17:36.607954 10638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:36.688001 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 10516
I20260620 15:17:36.704557 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 10165
I20260620 15:17:36.705660 10226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:36.803067 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 10165
2026-06-20T15:17:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2222 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:17:36 dist-test-slave-7s64 krb5kdc[10661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:17:36 dist-test-slave-7s64 krb5kdc[10661](info): set up 2 sockets
Jun 20 15:17:36 dist-test-slave-7s64 krb5kdc[10661](info): commencing operation
krb5kdc: starting...
W20260620 15:17:37.893524 389 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:17:38 dist-test-slave-7s64 krb5kdc[10661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:17:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:17:38.102142 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36719
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:38.429514 10674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:38.429760 10674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:38.429908 10674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:38.436941 10674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:17:38.437032 10674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:38.437212 10674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:38.437342 10674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:17:38.437422 10674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:17:38.446682 10674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:38.447860 10674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:38.449141 10674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:38.460736 10682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:38.460780 10679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:38.460805 10680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:38.461632 10674 server_base.cc:1034] running on GCE node
I20260620 15:17:38.462388 10674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:38.463667 10674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:38.464879 10674 hybrid_clock.cc:648] HybridClock initialized: now 1781968658464836 us; error 62 us; skew 500 ppm
I20260620 15:17:38.465198 10674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:38 dist-test-slave-7s64 krb5kdc[10661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:38.470659 10674 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:17:38.472047 10674 webserver.cc:460] Webserver started at http://127.0.97.126:35339/ using document root <none> and password file <none>
I20260620 15:17:38.472738 10674 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:38.472869 10674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:38.473146 10674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:38.475854 10674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "fd9b1fdc8a374cce9111ecf759564013"
format_stamp: "Formatted at 2026-06-20 15:17:38 on dist-test-slave-7s64"
server_key: "178be608a1e479e2e219b9b5e98ef5e9"
server_key_iv: "f6128d76bdfdb93f42a1a4ea162da5c7"
server_key_version: "encryptionkey@0"
I20260620 15:17:38.476626 10674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "fd9b1fdc8a374cce9111ecf759564013"
format_stamp: "Formatted at 2026-06-20 15:17:38 on dist-test-slave-7s64"
server_key: "178be608a1e479e2e219b9b5e98ef5e9"
server_key_iv: "f6128d76bdfdb93f42a1a4ea162da5c7"
server_key_version: "encryptionkey@0"
I20260620 15:17:38.481356 10674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:17:38.484855 10689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:38.486140 10674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:17:38.486321 10674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "fd9b1fdc8a374cce9111ecf759564013"
format_stamp: "Formatted at 2026-06-20 15:17:38 on dist-test-slave-7s64"
server_key: "178be608a1e479e2e219b9b5e98ef5e9"
server_key_iv: "f6128d76bdfdb93f42a1a4ea162da5c7"
server_key_version: "encryptionkey@0"
I20260620 15:17:38.486531 10674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:38.497653 10674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:38.501506 10674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:38.501773 10674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:38.519397 10674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36719
I20260620 15:17:38.519428 10740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36719 every 8 connection(s)
I20260620 15:17:38.520720 10674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:17:38.524351 10741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:38.530642 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 10674
I20260620 15:17:38.530843 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:17:38.531607 389 external_mini_cluster.cc:1428] Setting key 3da1cc228bce53c8c833939fc3a4dfc3
I20260620 15:17:38.535287 10741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013: Bootstrap starting.
I20260620 15:17:38.538278 10741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:38.539283 10741 log.cc:824] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:38.541872 10741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013: No bootstrap required, opened a new log
Jun 20 15:17:38 dist-test-slave-7s64 krb5kdc[10661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:17:38.551366 10741 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9b1fdc8a374cce9111ecf759564013" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36719 } }
I20260620 15:17:38.551818 10741 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:38.551934 10741 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9b1fdc8a374cce9111ecf759564013, State: Initialized, Role: FOLLOWER
I20260620 15:17:38.552650 10741 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9b1fdc8a374cce9111ecf759564013" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36719 } }
I20260620 15:17:38.552913 10741 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:17:38.553046 10741 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:17:38.553229 10741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:38.554114 10744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:38.533655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.0.97.126:36719)
0620 15:17:38.534334 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:38.534346 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:38.534383 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:17:38.536001 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:38.536021 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:38.536368 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:38.536798 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:38.538166 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:38.539520 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:38.540803 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:38.541434 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:38.541630 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:38.545645 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:38.545658 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:38.545676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:38.545723 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:38.549700 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:38.550350 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:38.550358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:38.550366 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:38.550552 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:38.550897 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:38.550903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:38.550910 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:38.551579 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:38.551929 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:38.552316 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 15:17:38.552768 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":289,"threads_started":1}
I20260620 15:17:38.556193 10741 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9b1fdc8a374cce9111ecf759564013" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36719 } }
I20260620 15:17:38.556780 10741 leader_election.cc:302] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd9b1fdc8a374cce9111ecf759564013; no voters:
I20260620 15:17:38.557348 10741 leader_election.cc:288] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:17:38.557471 10746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:38.558164 10746 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [term 1 LEADER]: Becoming Leader. State: Replica: fd9b1fdc8a374cce9111ecf759564013, State: Running, Role: LEADER
I20260620 15:17:38.558679 10746 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9b1fdc8a374cce9111ecf759564013" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36719 } }
I20260620 15:17:38.559248 10741 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:17:38.561496 10747 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd9b1fdc8a374cce9111ecf759564013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9b1fdc8a374cce9111ecf759564013" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36719 } } }
I20260620 15:17:38.561971 10747 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [sys.catalog]: This master's current role is: LEADER
I20260620 15:17:38.563607 10748 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd9b1fdc8a374cce9111ecf759564013. Latest consensus state: current_term: 1 leader_uuid: "fd9b1fdc8a374cce9111ecf759564013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9b1fdc8a374cce9111ecf759564013" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36719 } } }
I20260620 15:17:38.564064 10748 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013 [sys.catalog]: This master's current role is: LEADER
W20260620 15:17:38.572960 10762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:17:38.573086 10762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:17:38.573246 10755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:17:38.575297 10755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:17:38.582616 10755 catalog_manager.cc:1347] Generated new cluster ID: 19a6b12d4e374a2080a2ab082d915b01
I20260620 15:17:38.582710 10755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:17:38.602138 10755 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:17:38.603333 10755 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:17:38.616940 10755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013: Generated new TSK 0
I20260620 15:17:38.617672 10755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:17:38.667322 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36719
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:38.982587 10767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:38.982895 10767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:38.983119 10767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:38.991194 10767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:38.991294 10767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:38.991468 10767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:17:39.001480 10767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:39.002677 10767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:39.003921 10767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:39.016209 10772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:39.016175 10775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:39.016825 10773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:39.017323 10767 server_base.cc:1034] running on GCE node
I20260620 15:17:39.017876 10767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:39.018690 10767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:39.019894 10767 hybrid_clock.cc:648] HybridClock initialized: now 1781968659019854 us; error 32 us; skew 500 ppm
I20260620 15:17:39.020205 10767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:39 dist-test-slave-7s64 krb5kdc[10661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:39.023288 10767 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:17:39.024751 10767 webserver.cc:460] Webserver started at http://127.0.97.65:37159/ using document root <none> and password file <none>
I20260620 15:17:39.025406 10767 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:39.025552 10767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:39.025861 10767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:39.028756 10767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "d3b48d5dd07844f88fa3ea36b02525a3"
format_stamp: "Formatted at 2026-06-20 15:17:39 on dist-test-slave-7s64"
server_key: "ce0df60ed017603e39d537a5d697423b"
server_key_iv: "4a2288453e8730f9a5ff00fa8d6d1fa2"
server_key_version: "encryptionkey@0"
I20260620 15:17:39.029529 10767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "d3b48d5dd07844f88fa3ea36b02525a3"
format_stamp: "Formatted at 2026-06-20 15:17:39 on dist-test-slave-7s64"
server_key: "ce0df60ed017603e39d537a5d697423b"
server_key_iv: "4a2288453e8730f9a5ff00fa8d6d1fa2"
server_key_version: "encryptionkey@0"
I20260620 15:17:39.034758 10767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:17:39.037849 10782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:39.039057 10767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:17:39.039211 10767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "d3b48d5dd07844f88fa3ea36b02525a3"
format_stamp: "Formatted at 2026-06-20 15:17:39 on dist-test-slave-7s64"
server_key: "ce0df60ed017603e39d537a5d697423b"
server_key_iv: "4a2288453e8730f9a5ff00fa8d6d1fa2"
server_key_version: "encryptionkey@0"
I20260620 15:17:39.039372 10767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:39.057714 10767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:39.061847 10767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:39.062121 10767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:39.062950 10767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:39.064200 10767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:39.064280 10767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:39.064419 10767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:39.064525 10767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:39.098186 10767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42257
I20260620 15:17:39.098256 10894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42257 every 8 connection(s)
I20260620 15:17:39.099479 10767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:17:39.105871 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 10767
I20260620 15:17:39.106060 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:17:39.106726 389 external_mini_cluster.cc:1428] Setting key e427dc24fa3d4a1413ff1d8ffcbd6811
Jun 20 15:17:39 dist-test-slave-7s64 krb5kdc[10661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:17:39.120599 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:39.102002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:39531 (local address 127.0.97.126:36719)
0620 15:17:39.102425 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:39.102430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:39.103382 (+ 952us) server_negotiation.cc:407] Connection header received
0620 15:17:39.105247 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:39.105251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:39.105325 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:39.105418 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:39.106997 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:39.107933 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:39.109820 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:39.110554 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:39.110705 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:39.113975 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:39.113984 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:39.113991 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:39.114035 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:39.117282 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:39.118196 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:39.118200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:39.118202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:39.118310 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:39.118818 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:39.118825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:39.118831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:39.119096 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:39.119212 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:39.120105 (+ 893us) server_negotiation.cc:299] Negotiation successful
0620 15:17:39.120296 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":206,"threads_started":1}
I20260620 15:17:39.122133 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:39.102536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36719 (local address 127.0.97.65:39531)
0620 15:17:39.103222 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:39.103259 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:39.105001 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:39.105660 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:39.105668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:39.106254 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:39.106795 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:39.106839 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:39.108098 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:39.108104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:39.109678 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:39.109691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:39.110760 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:39.110766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:39.110961 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:39.111678 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:39.111702 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:39.113766 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:39.117409 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:39.117421 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:39.117439 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:39.118071 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:39.118412 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:39.118419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:39.118426 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:39.118672 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:39.119240 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:39.119251 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:39.119687 (+ 436us) client_negotiation.cc:769] Sending connection context
0620 15:17:39.120029 (+ 342us) client_negotiation.cc:240] Negotiation successful
0620 15:17:39.120623 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":255,"threads_started":1}
I20260620 15:17:39.123595 10895 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36719
I20260620 15:17:39.123881 10895 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:39.124583 10895 heartbeater.cc:511] Master 127.0.97.126:36719 requested a full tablet report, sending...
I20260620 15:17:39.126581 10706 ts_manager.cc:194] Registered new tserver with Master: d3b48d5dd07844f88fa3ea36b02525a3 (127.0.97.65:42257)
I20260620 15:17:39.127895 10706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:39531
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:17:39.158963 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36719
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:39.463354 10901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:39.463589 10901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:39.463742 10901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:39.470644 10901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:39.470731 10901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:39.470920 10901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:17:39.480801 10901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:39.482029 10901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:39.483390 10901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:39.496237 10906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:39.496201 10907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:39.496279 10909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:39.497104 10901 server_base.cc:1034] running on GCE node
I20260620 15:17:39.497699 10901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:39.498474 10901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:39.499686 10901 hybrid_clock.cc:648] HybridClock initialized: now 1781968659499651 us; error 46 us; skew 500 ppm
I20260620 15:17:39.499980 10901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:39 dist-test-slave-7s64 krb5kdc[10661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:39.502923 10901 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:17:39.504262 10901 webserver.cc:460] Webserver started at http://127.0.97.66:41775/ using document root <none> and password file <none>
I20260620 15:17:39.504951 10901 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:39.505079 10901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:39.505352 10901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:39.508020 10901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "c39403b5750340fcbcc8b5539694c873"
format_stamp: "Formatted at 2026-06-20 15:17:39 on dist-test-slave-7s64"
server_key: "305f7ee1bc5947287e96e2d48c215c9c"
server_key_iv: "24cb3353995e1dea8373bfb2b96bf1af"
server_key_version: "encryptionkey@0"
I20260620 15:17:39.508782 10901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "c39403b5750340fcbcc8b5539694c873"
format_stamp: "Formatted at 2026-06-20 15:17:39 on dist-test-slave-7s64"
server_key: "305f7ee1bc5947287e96e2d48c215c9c"
server_key_iv: "24cb3353995e1dea8373bfb2b96bf1af"
server_key_version: "encryptionkey@0"
I20260620 15:17:39.513427 10901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:17:39.516537 10916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:39.517771 10901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 15:17:39.517921 10901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "c39403b5750340fcbcc8b5539694c873"
format_stamp: "Formatted at 2026-06-20 15:17:39 on dist-test-slave-7s64"
server_key: "305f7ee1bc5947287e96e2d48c215c9c"
server_key_iv: "24cb3353995e1dea8373bfb2b96bf1af"
server_key_version: "encryptionkey@0"
I20260620 15:17:39.518103 10901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:39.540237 10901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:39.544258 10901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:39.544559 10901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:39.545735 10901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:39.546962 10901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:39.547060 10901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:39.547202 10901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:39.547269 10901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:39.579483 10901 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:36157
I20260620 15:17:39.579536 11028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:36157 every 8 connection(s)
I20260620 15:17:39.580783 10901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:17:39.586483 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 10901
I20260620 15:17:39.586674 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:17:39.587203 389 external_mini_cluster.cc:1428] Setting key 1a7554cb96736d0254bcc8fea60b76b6
Jun 20 15:17:39 dist-test-slave-7s64 krb5kdc[10661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:17:39.602303 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:39.583105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:37451 (local address 127.0.97.126:36719)
0620 15:17:39.583270 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:39.583274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:39.584371 (+ 1097us) server_negotiation.cc:407] Connection header received
0620 15:17:39.586071 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:39.586076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:39.586140 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:39.586226 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:39.588074 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:39.589274 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:39.591681 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:39.592103 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:39.592202 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:39.596110 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:39.596115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:39.596118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:39.596148 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:39.599291 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:39.599956 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:39.599964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:39.599970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:39.600114 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:39.600506 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:39.600514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:39.600520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:39.600872 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:39.601017 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:39.601638 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 15:17:39.601878 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:17:39.602856 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:39.583574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36719 (local address 127.0.97.66:37451)
0620 15:17:39.584211 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:39.584250 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:39.585830 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:39.586488 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:39.586501 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:39.587190 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:39.587889 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:39.587908 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:39.589421 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:39.589426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:39.591542 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:39.591559 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:39.592309 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:39.592317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:39.592636 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:39.593398 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:39.593429 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:39.595910 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:39.599422 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:39.599428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:39.599438 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:39.599846 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:39.600225 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:39.600228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:39.600230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:39.600388 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:39.601003 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:39.601009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:39.601289 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 15:17:39.601509 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 15:17:39.601854 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":238,"threads_started":1}
I20260620 15:17:39.604064 11029 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36719
I20260620 15:17:39.604336 11029 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:39.604992 11029 heartbeater.cc:511] Master 127.0.97.126:36719 requested a full tablet report, sending...
I20260620 15:17:39.606312 10706 ts_manager.cc:194] Registered new tserver with Master: c39403b5750340fcbcc8b5539694c873 (127.0.97.66:36157)
I20260620 15:17:39.607107 10706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:37451
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:17:39.635341 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36719
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:17:39.945933 11034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:17:39.946208 11034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:17:39.946355 11034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:17:39.953229 11034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:17:39.953356 11034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:17:39.953509 11034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:17:39.962754 11034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:17:39.963904 11034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:17:39.965186 11034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:17:39.978401 11039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:39.978451 11042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:17:39.978482 11040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:17:39.979048 11034 server_base.cc:1034] running on GCE node
I20260620 15:17:39.979734 11034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:17:39.980690 11034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:17:39.981901 11034 hybrid_clock.cc:648] HybridClock initialized: now 1781968659981857 us; error 42 us; skew 500 ppm
I20260620 15:17:39.982218 11034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:17:39 dist-test-slave-7s64 krb5kdc[10661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:17:39.985260 11034 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:17:39.986562 11034 webserver.cc:460] Webserver started at http://127.0.97.67:43641/ using document root <none> and password file <none>
I20260620 15:17:39.987175 11034 fs_manager.cc:359] Metadata directory not provided
I20260620 15:17:39.987267 11034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:17:39.987494 11034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:17:39.990492 11034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "16872cc4776d407e9bbbf9a8eaab06ef"
format_stamp: "Formatted at 2026-06-20 15:17:39 on dist-test-slave-7s64"
server_key: "a30140a95f889c4d25f0571ba3ede2da"
server_key_iv: "cfd550c62b7e95f027c0d5f749a7a7bf"
server_key_version: "encryptionkey@0"
I20260620 15:17:39.991237 11034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "16872cc4776d407e9bbbf9a8eaab06ef"
format_stamp: "Formatted at 2026-06-20 15:17:39 on dist-test-slave-7s64"
server_key: "a30140a95f889c4d25f0571ba3ede2da"
server_key_iv: "cfd550c62b7e95f027c0d5f749a7a7bf"
server_key_version: "encryptionkey@0"
I20260620 15:17:39.996186 11034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:17:39.999447 11049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:40.000828 11034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:17:40.000952 11034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "16872cc4776d407e9bbbf9a8eaab06ef"
format_stamp: "Formatted at 2026-06-20 15:17:39 on dist-test-slave-7s64"
server_key: "a30140a95f889c4d25f0571ba3ede2da"
server_key_iv: "cfd550c62b7e95f027c0d5f749a7a7bf"
server_key_version: "encryptionkey@0"
I20260620 15:17:40.001096 11034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:17:40.015252 11034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:17:40.019157 11034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:17:40.019371 11034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:17:40.020182 11034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:17:40.021495 11034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:17:40.021574 11034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:40.021652 11034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:17:40.021699 11034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:17:40.053913 11034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:33095
I20260620 15:17:40.053997 11161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:33095 every 8 connection(s)
I20260620 15:17:40.055223 11034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:17:40.063071 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 11034
I20260620 15:17:40.063258 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:17:40.063679 389 external_mini_cluster.cc:1428] Setting key 892b6a8375a2b6670fda7d3189c7c8f0
Jun 20 15:17:40 dist-test-slave-7s64 krb5kdc[10661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:17:40.073910 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.057645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59699 (local address 127.0.97.126:36719)
0620 15:17:40.057832 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.057837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.058911 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 15:17:40.060758 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.060763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.060847 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.060965 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:40.062350 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.063101 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.064883 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.065337 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.065466 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.069088 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:40.069092 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:40.069096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:40.069128 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:40.071508 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:40.072129 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:40.072132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:40.072135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:40.072237 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:40.072629 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:40.072632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:40.072634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:40.072871 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:40.072975 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.073520 (+ 545us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.073680 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:17:40.074770 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.058050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36719 (local address 127.0.97.67:59699)
0620 15:17:40.058741 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:40.058802 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:40.060459 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:40.061184 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:40.061192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:40.061657 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:17:40.062156 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.062172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.063253 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.063257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.064732 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.064749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.065577 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.065598 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.065884 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.066610 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:17:40.066642 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:17:40.068899 (+ 2257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:17:40.071622 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:40.071628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:40.071638 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:40.072009 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:40.072328 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:17:40.072331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:17:40.072333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:17:40.072532 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:17:40.072973 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:17:40.072978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:17:40.073248 (+ 270us) client_negotiation.cc:769] Sending connection context
0620 15:17:40.073447 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 15:17:40.073804 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":264,"threads_started":1}
I20260620 15:17:40.075953 11162 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36719
I20260620 15:17:40.076215 11162 heartbeater.cc:464] Registering TS with master...
I20260620 15:17:40.076887 11162 heartbeater.cc:511] Master 127.0.97.126:36719 requested a full tablet report, sending...
I20260620 15:17:40.078336 10706 ts_manager.cc:194] Registered new tserver with Master: 16872cc4776d407e9bbbf9a8eaab06ef (127.0.97.67:33095)
I20260620 15:17:40.078811 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:17:40.079213 10706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:59699
I20260620 15:17:40.094934 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.083612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.0.97.126:36719)
0620 15:17:40.083803 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.083811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.084156 (+ 345us) server_negotiation.cc:407] Connection header received
0620 15:17:40.084276 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.084279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.084329 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.084404 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:40.085741 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.086400 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.087560 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.087955 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.088049 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.090064 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:40.090068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:40.090072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:40.090097 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:40.092584 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:40.093207 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:40.093211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:40.093213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:40.093350 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:40.093670 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:40.093673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:40.093675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:40.093965 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:40.094151 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.094411 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.094565 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:17:40.098835 10706 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:17:40.105546 10706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:17:40.131266 10895 heartbeater.cc:503] Master 127.0.97.126:36719 was elected leader, sending a full tablet report...
I20260620 15:17:40.138095 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.116911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.0.97.67:33095)
0620 15:17:40.129946 (+ 13035us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.129953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.129980 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:17:40.130070 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.130080 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.130317 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.130510 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:40.131257 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.133211 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.135579 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.136967 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.137111 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.137346 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.137597 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.137756 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12859,"thread_start_us":12932,"threads_started":1}
I20260620 15:17:40.138092 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.116723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:33095 (local address 127.0.0.1:53308)
0620 15:17:40.128797 (+ 12074us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:40.128819 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:40.128917 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:40.130525 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:40.130533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:40.130565 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:40.130989 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.131004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.133381 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.133387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.135411 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.135427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.137149 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.137155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.137350 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.137398 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 15:17:40.137534 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 15:17:40.137696 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11959,"mutex_wait_us":86,"thread_start_us":11126,"threads_started":1}
I20260620 15:17:40.139245 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.119421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.0.97.66:36157)
0620 15:17:40.129467 (+ 10046us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.129478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.129941 (+ 463us) server_negotiation.cc:407] Connection header received
0620 15:17:40.130214 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.130221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.130398 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.130528 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:40.131455 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.133471 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.136248 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.137725 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.137851 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.138054 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.138764 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.138926 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9887,"thread_start_us":9760,"threads_started":1}
I20260620 15:17:40.139281 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.118749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:36157 (local address 127.0.0.1:45432)
0620 15:17:40.129835 (+ 11086us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:40.129858 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:40.130035 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:40.130813 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:40.130816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:40.130834 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:40.131248 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.131262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.133589 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.133597 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.136077 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.136092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.138355 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.138363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.138546 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.138592 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 15:17:40.138691 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:17:40.138820 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10956,"mutex_wait_us":81,"thread_start_us":10814,"threads_started":1}
I20260620 15:17:40.139739 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.116545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42257 (local address 127.0.0.1:38514)
0620 15:17:40.128185 (+ 11640us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:40.128224 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:40.128381 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:40.131563 (+ 3182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:40.131568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:40.131596 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:40.132054 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.132068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.134148 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.134153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.136869 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.136881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.138861 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.138869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.139036 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.139069 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:17:40.139167 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:17:40.139305 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11413,"mutex_wait_us":23,"thread_start_us":12004,"threads_started":1}
I20260620 15:17:40.139894 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.117055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.0.97.65:42257)
0620 15:17:40.130627 (+ 13572us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.130637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.130669 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:17:40.130768 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.130776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.130999 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.131191 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:40.132282 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.133869 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.137009 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.138434 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.138547 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.138723 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.139399 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.139561 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13325,"thread_start_us":13160,"threads_started":1}
I20260620 15:17:40.142086 11097 tablet_service.cc:1467] Processing CreateTablet for tablet f0239aed13344305a189da98f31d1914 (DEFAULT_TABLE table=test-table [id=395649c30c8549e0928527b9a232eeef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:40.142866 10830 tablet_service.cc:1467] Processing CreateTablet for tablet f0239aed13344305a189da98f31d1914 (DEFAULT_TABLE table=test-table [id=395649c30c8549e0928527b9a232eeef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:40.143527 11097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0239aed13344305a189da98f31d1914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:40.143934 10830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0239aed13344305a189da98f31d1914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:40.147732 10964 tablet_service.cc:1467] Processing CreateTablet for tablet f0239aed13344305a189da98f31d1914 (DEFAULT_TABLE table=test-table [id=395649c30c8549e0928527b9a232eeef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:17:40.148972 10964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0239aed13344305a189da98f31d1914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:17:40.155166 11180 tablet_bootstrap.cc:492] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: Bootstrap starting.
I20260620 15:17:40.157320 11181 tablet_bootstrap.cc:492] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: Bootstrap starting.
I20260620 15:17:40.159601 11180 tablet_bootstrap.cc:654] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:40.160881 11181 tablet_bootstrap.cc:654] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:40.160993 11180 log.cc:824] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:40.161944 11182 tablet_bootstrap.cc:492] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: Bootstrap starting.
I20260620 15:17:40.162281 11181 log.cc:824] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:40.164929 11180 tablet_bootstrap.cc:492] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: No bootstrap required, opened a new log
I20260620 15:17:40.165189 11180 ts_tablet_manager.cc:1397] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 15:17:40.165367 11181 tablet_bootstrap.cc:492] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: No bootstrap required, opened a new log
I20260620 15:17:40.165365 11182 tablet_bootstrap.cc:654] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: Neither blocks nor log segments found. Creating new log.
I20260620 15:17:40.165611 11181 ts_tablet_manager.cc:1397] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:17:40.166476 11182 log.cc:824] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: Log is configured to *not* fsync() on all Append() calls
I20260620 15:17:40.169669 11182 tablet_bootstrap.cc:492] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: No bootstrap required, opened a new log
I20260620 15:17:40.169903 11182 ts_tablet_manager.cc:1397] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 15:17:40.173668 11180 raft_consensus.cc:348] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 } } peers { permanent_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33095 } } peers { permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 } }
I20260620 15:17:40.174036 11180 raft_consensus.cc:374] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:40.174144 11180 raft_consensus.cc:729] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16872cc4776d407e9bbbf9a8eaab06ef, State: Initialized, Role: FOLLOWER
I20260620 15:17:40.174750 11180 consensus_queue.cc:260] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 } } peers { permanent_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33095 } } peers { permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 } }
I20260620 15:17:40.175819 11162 heartbeater.cc:503] Master 127.0.97.126:36719 was elected leader, sending a full tablet report...
I20260620 15:17:40.175693 11181 raft_consensus.cc:348] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 } } peers { permanent_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33095 } } peers { permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 } }
I20260620 15:17:40.176057 11181 raft_consensus.cc:374] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:40.176192 11181 raft_consensus.cc:729] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b48d5dd07844f88fa3ea36b02525a3, State: Initialized, Role: FOLLOWER
I20260620 15:17:40.176679 11180 ts_tablet_manager.cc:1428] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260620 15:17:40.176759 11181 consensus_queue.cc:260] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 } } peers { permanent_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33095 } } peers { permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 } }
I20260620 15:17:40.179210 11181 ts_tablet_manager.cc:1428] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 15:17:40.181195 11182 raft_consensus.cc:348] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 } } peers { permanent_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33095 } } peers { permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 } }
I20260620 15:17:40.181557 11182 raft_consensus.cc:374] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:17:40.181661 11182 raft_consensus.cc:729] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39403b5750340fcbcc8b5539694c873, State: Initialized, Role: FOLLOWER
I20260620 15:17:40.182200 11182 consensus_queue.cc:260] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 } } peers { permanent_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33095 } } peers { permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 } }
I20260620 15:17:40.183239 11029 heartbeater.cc:503] Master 127.0.97.126:36719 was elected leader, sending a full tablet report...
I20260620 15:17:40.183847 11182 ts_tablet_manager.cc:1428] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 15:17:40.188930 11186 raft_consensus.cc:482] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:17:40.189109 11186 raft_consensus.cc:504] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 } } peers { permanent_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33095 } } peers { permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 } }
I20260620 15:17:40.190783 11186 leader_election.cc:288] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3b48d5dd07844f88fa3ea36b02525a3 (127.0.97.65:42257), c39403b5750340fcbcc8b5539694c873 (127.0.97.66:36157)
I20260620 15:17:40.197152 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.191163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:36659 (local address 127.0.97.66:36157)
0620 15:17:40.191332 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.191338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.191400 (+ 62us) server_negotiation.cc:407] Connection header received
0620 15:17:40.191562 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.191565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.191619 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.191690 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:40.192180 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.193253 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.195112 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.196342 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.196432 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.196544 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.196880 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.196962 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:17:40.197263 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.191148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:36157 (local address 127.0.97.67:36659)
0620 15:17:40.191330 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:40.191350 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:40.191460 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:40.191746 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:40.191753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:40.191788 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:40.192034 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.192055 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.193371 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.193376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.194963 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.194973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.196510 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.196518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.196726 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.196761 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:17:40.196862 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:17:40.196995 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":1}
I20260620 15:17:40.198275 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.191161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42257 (local address 127.0.97.67:41327)
0620 15:17:40.192174 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0620 15:17:40.192191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:17:40.192314 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:17:40.192615 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:17:40.192621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:17:40.192647 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:17:40.192971 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.192980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.194340 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.194343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.196114 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:17:40.196129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.197679 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.197685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:17:40.197855 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.197878 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:17:40.197950 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 15:17:40.198027 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":371,"thread_start_us":357,"threads_started":1}
I20260620 15:17:40.198359 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.191259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:41327 (local address 127.0.97.65:42257)
0620 15:17:40.191427 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.191432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.192273 (+ 841us) server_negotiation.cc:407] Connection header received
0620 15:17:40.192413 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.192416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.192467 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.192583 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:17:40.193146 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.194181 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.196275 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.197507 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.197617 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.197682 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.197998 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.198125 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:17:40.198225 10984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0239aed13344305a189da98f31d1914" candidate_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39403b5750340fcbcc8b5539694c873" is_pre_election: true
I20260620 15:17:40.198640 10984 raft_consensus.cc:2454] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16872cc4776d407e9bbbf9a8eaab06ef in term 0.
I20260620 15:17:40.199280 11051 leader_election.cc:302] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16872cc4776d407e9bbbf9a8eaab06ef, c39403b5750340fcbcc8b5539694c873; no voters:
I20260620 15:17:40.199391 10850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0239aed13344305a189da98f31d1914" candidate_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" is_pre_election: true
I20260620 15:17:40.199671 11186 raft_consensus.cc:2781] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:17:40.199815 11186 raft_consensus.cc:482] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:17:40.199915 11186 raft_consensus.cc:3037] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:40.199888 10850 raft_consensus.cc:2454] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16872cc4776d407e9bbbf9a8eaab06ef in term 0.
I20260620 15:17:40.202105 11186 raft_consensus.cc:504] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 } } peers { permanent_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33095 } } peers { permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 } }
I20260620 15:17:40.202633 11186 leader_election.cc:288] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [CANDIDATE]: Term 1 election: Requested vote from peers d3b48d5dd07844f88fa3ea36b02525a3 (127.0.97.65:42257), c39403b5750340fcbcc8b5539694c873 (127.0.97.66:36157)
I20260620 15:17:40.202977 10850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0239aed13344305a189da98f31d1914" candidate_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b48d5dd07844f88fa3ea36b02525a3"
I20260620 15:17:40.203001 10984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0239aed13344305a189da98f31d1914" candidate_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39403b5750340fcbcc8b5539694c873"
I20260620 15:17:40.203110 10850 raft_consensus.cc:3037] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:40.203186 10984 raft_consensus.cc:3037] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:17:40.205377 10850 raft_consensus.cc:2454] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16872cc4776d407e9bbbf9a8eaab06ef in term 1.
I20260620 15:17:40.205506 10984 raft_consensus.cc:2454] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16872cc4776d407e9bbbf9a8eaab06ef in term 1.
I20260620 15:17:40.205761 11052 leader_election.cc:302] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16872cc4776d407e9bbbf9a8eaab06ef, d3b48d5dd07844f88fa3ea36b02525a3; no voters:
I20260620 15:17:40.206061 11186 raft_consensus.cc:2781] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:17:40.206511 11186 raft_consensus.cc:686] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 1 LEADER]: Becoming Leader. State: Replica: 16872cc4776d407e9bbbf9a8eaab06ef, State: Running, Role: LEADER
I20260620 15:17:40.206969 11186 consensus_queue.cc:237] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 } } peers { permanent_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33095 } } peers { permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 } }
I20260620 15:17:40.211083 10705 catalog_manager.cc:5495] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 16872cc4776d407e9bbbf9a8eaab06ef (127.0.97.67). New cstate: current_term: 1 leader_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16872cc4776d407e9bbbf9a8eaab06ef" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 33095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 } health_report { overall_health: UNKNOWN } } }
I20260620 15:17:40.237128 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.232350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.0.97.67:33095)
0620 15:17:40.232524 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.232528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.232545 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:17:40.232729 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.232733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.232801 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.232921 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:17:40.233263 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.233922 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.235214 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.235590 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.235679 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.236093 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:17:40.236616 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:17:40.236721 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.236893 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.236947 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:17:40.244752 10984 raft_consensus.cc:1261] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873 [term 1 FOLLOWER]: Refusing update from remote peer 16872cc4776d407e9bbbf9a8eaab06ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:40.244755 10850 raft_consensus.cc:1261] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3 [term 1 FOLLOWER]: Refusing update from remote peer 16872cc4776d407e9bbbf9a8eaab06ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:17:40.245669 11190 consensus_queue.cc:1035] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39403b5750340fcbcc8b5539694c873" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:40.246053 11186 consensus_queue.cc:1035] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3b48d5dd07844f88fa3ea36b02525a3" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:17:40.258635 11194 mvcc.cc:205] Tried to move back new op lower bound from 7298943632353132544 to 7298943632210944000. Current Snapshot: MvccSnapshot[applied={T|T < 7298943632353132544}]
I20260620 15:17:40.277253 10705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:40.277489 10705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:17:40.280673 10705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd9b1fdc8a374cce9111ecf759564013: Sending DeleteTablet for 3 replicas of tablet f0239aed13344305a189da98f31d1914
I20260620 15:17:40.281708 10830 tablet_service.cc:1514] Processing DeleteTablet for tablet f0239aed13344305a189da98f31d1914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:40 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:38514
I20260620 15:17:40.281715 11097 tablet_service.cc:1514] Processing DeleteTablet for tablet f0239aed13344305a189da98f31d1914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:40 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:53308
I20260620 15:17:40.282084 10964 tablet_service.cc:1514] Processing DeleteTablet for tablet f0239aed13344305a189da98f31d1914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:17:40 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:45432
I20260620 15:17:40.284452 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 10767
I20260620 15:17:40.291831 11205 tablet_replica.cc:330] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: stopping tablet replica
I20260620 15:17:40.292395 11204 tablet_replica.cc:330] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: stopping tablet replica
I20260620 15:17:40.292665 11205 raft_consensus.cc:2229] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:40.292759 11204 raft_consensus.cc:2229] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:17:40.293241 11205 raft_consensus.cc:2258] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:40.293298 11204 raft_consensus.cc:2258] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:40.294531 11203 tablet_replica.cc:330] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: stopping tablet replica
I20260620 15:17:40.294842 11203 raft_consensus.cc:2229] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:17:40.295215 11203 raft_consensus.cc:2258] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:17:40.295742 11205 ts_tablet_manager.cc:1905] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:40.295742 11204 ts_tablet_manager.cc:1905] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:40.298137 11203 ts_tablet_manager.cc:1905] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:17:40.302682 11204 ts_tablet_manager.cc:1918] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:40.302832 11204 log.cc:1196] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/f0239aed13344305a189da98f31d1914
I20260620 15:17:40.303335 11204 ts_tablet_manager.cc:1939] T f0239aed13344305a189da98f31d1914 P 16872cc4776d407e9bbbf9a8eaab06ef: Deleting consensus metadata
Jun 20 15:17:40 dist-test-slave-7s64 krb5kdc[10661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:17:40.304919 10690 catalog_manager.cc:4841] TS 16872cc4776d407e9bbbf9a8eaab06ef (127.0.97.67:33095): tablet f0239aed13344305a189da98f31d1914 (table test-table [id=395649c30c8549e0928527b9a232eeef]) successfully deleted
I20260620 15:17:40.306614 11203 ts_tablet_manager.cc:1918] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:40.306762 11203 log.cc:1196] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/f0239aed13344305a189da98f31d1914
I20260620 15:17:40.307250 11203 ts_tablet_manager.cc:1939] T f0239aed13344305a189da98f31d1914 P d3b48d5dd07844f88fa3ea36b02525a3: Deleting consensus metadata
I20260620 15:17:40.308853 10692 catalog_manager.cc:4841] TS d3b48d5dd07844f88fa3ea36b02525a3 (127.0.97.65:42257): tablet f0239aed13344305a189da98f31d1914 (table test-table [id=395649c30c8549e0928527b9a232eeef]) successfully deleted
I20260620 15:17:40.309474 11205 ts_tablet_manager.cc:1918] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:17:40.309630 11205 log.cc:1196] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/f0239aed13344305a189da98f31d1914
I20260620 15:17:40.310149 11205 ts_tablet_manager.cc:1939] T f0239aed13344305a189da98f31d1914 P c39403b5750340fcbcc8b5539694c873: Deleting consensus metadata
I20260620 15:17:40.311611 10691 catalog_manager.cc:4841] TS c39403b5750340fcbcc8b5539694c873 (127.0.97.66:36157): tablet f0239aed13344305a189da98f31d1914 (table test-table [id=395649c30c8549e0928527b9a232eeef]) successfully deleted
I20260620 15:17:40.312786 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.285818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.0.97.65:42257)
0620 15:17:40.286029 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.286039 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.293982 (+ 7943us) server_negotiation.cc:407] Connection header received
0620 15:17:40.294319 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.294325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.294429 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.294509 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:40.296845 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.297997 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.299867 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.300565 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.300665 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.305872 (+ 5207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:40.305885 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:40.305902 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:40.305939 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:40.309837 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:40.310630 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:40.310639 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:40.310643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:40.310786 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:40.311210 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:40.311217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:40.311220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:40.311624 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:40.311906 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.312191 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.312406 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:17:40.313503 10890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:40.401381 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 10767
I20260620 15:17:40.420863 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 10901
Jun 20 15:17:40 dist-test-slave-7s64 krb5kdc[10661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:17:40.436815 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.421729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.0.97.66:36157)
0620 15:17:40.421888 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.421892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.421931 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:17:40.422098 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.422103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.422174 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.422254 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:40.423795 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.424553 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.426319 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.426705 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.426790 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.431457 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:40.431470 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:40.431484 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:40.431513 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:40.434238 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:40.434839 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:40.434846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:40.434849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:40.434965 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:40.435305 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:40.435312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:40.435316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:40.435654 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:40.435924 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.436223 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.436428 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:17:40.437422 11024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:40.525832 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 10901
I20260620 15:17:40.543613 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 11034
Jun 20 15:17:40 dist-test-slave-7s64 krb5kdc[10661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:17:40.560328 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:17:40.544541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.0.97.67:33095)
0620 15:17:40.544719 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:17:40.544727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:17:40.544757 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:17:40.544988 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:17:40.544994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:17:40.545073 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:17:40.545188 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:17:40.546490 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.547392 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.548751 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:17:40.549263 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:17:40.549400 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:17:40.553873 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:17:40.553886 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:17:40.553905 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:17:40.553938 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:17:40.557392 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:40.558059 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:40.558070 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:40.558073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:40.558224 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:17:40.558633 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:17:40.558644 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:17:40.558646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:17:40.559059 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:17:40.559395 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 15:17:40.559682 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 15:17:40.559922 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:17:40.561115 11157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:40.694101 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 11034
I20260620 15:17:40.716336 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 10674
I20260620 15:17:40.717485 10736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:17:40.825067 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 10674
2026-06-20T15:17:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4023 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:17:40 dist-test-slave-7s64 krb5kdc[11213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:17:40 dist-test-slave-7s64 krb5kdc[11213](info): set up 2 sockets
Jun 20 15:17:40 dist-test-slave-7s64 krb5kdc[11213](info): commencing operation
krb5kdc: starting...
W20260620 15:17:41.921357 389 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:17:42 dist-test-slave-7s64 krb5kdc[11213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:17:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:17:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
I20260620 15:17:42.203511 389 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRnfwZV/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/postgres -l logfile start
2026-06-20 15:17:44.536 UTC [11243] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 15:17:44.536 UTC [11243] LOG: listening on IPv4 address "127.0.97.84", port 56819
2026-06-20 15:17:44.539 UTC [11243] LOG: listening on Unix socket "/tmp/.s.PGSQL.56819"
2026-06-20 15:17:44.561 UTC [11246] LOG: database system was shut down at 2026-06-20 15:17:43 UTC
2026-06-20 15:17:44.565 UTC [11243] LOG: database system is ready to accept connections
I20260620 15:17:45.537070 389 mini_postgres.cc:96] Postgres bound to 56819
2026-06-20 15:17:45.554 UTC [11254] FATAL: database "slave" does not exist
127.0.97.84:56819 - accepting connections
I20260620 15:17:45.555014 389 mini_ranger.cc:161] Starting Ranger...
I20260620 15:17:45.579025 389 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 15:17:46.041319 389 mini_ranger.cc:87] Created ranger Postgres database
I20260620 15:17:46.041481 389 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-admin
2026-06-20 15:17:46,334 [I] DB FLAVOR :POSTGRES
2026-06-20 15:17:46,334 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:17:46,334 [I] Checking connection..
2026-06-20 15:17:46,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 15:17:46,586 [I] Checking connection passed.
2026-06-20 15:17:46,586 [I] --------- Verifying version history table ---------
2026-06-20 15:17:46,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:17:46,822 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:17:46,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:17:47,059 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:17:47,059 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 15:17:47,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 15:17:47.310 UTC [11318] WARNING: there is no transaction in progress
2026-06-20 15:17:47,315 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 15:17:47,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:17:47,570 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 15:17:47,571 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:17:47,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 15:17:47,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 15:17:48,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7s64.c.gcp-upstream.internal','N') ;"
2026-06-20 15:17:48,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:17:48,552 [I] Table x_portal_user does not exist in database ranger
2026-06-20 15:17:48,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:17:48,784 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 15:17:48,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:17:49,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:17:49,255 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 15:17:49,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 15:17:50.146 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.162 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.178 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.192 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.207 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.224 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.275 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.281 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.295 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.305 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.314 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.326 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.335 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.341 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.347 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50.354 UTC [11426] WARNING: there is no transaction in progress
2026-06-20 15:17:50,956 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 15:17:50,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 15:17:51,202 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 15:17:51,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:17:51,435 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 15:17:51,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:17:51,688 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 15:17:51,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:17:51,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:17:52,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7s64.c.gcp-upstream.internal';"
2026-06-20 15:17:52,347 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 15:17:52,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 15:17:52,571 [I] DB_PATCHES have already been applied
I20260620 15:17:52.577454 389 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRnfwZV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 15:17:52.577554 389 mini_ranger.cc:193] Using host: 127.0.97.84
I20260620 15:17:52.591499 389 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.84:56827
Jun 20, 2026 3:17:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:17:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:17:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:17:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:17:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:17:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:17:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 15:17:53.716391 389 mini_ranger.cc:160] Time spent starting Ranger: real 8.161s user 0.002s sys 0.033s
Jun 20, 2026 3:18:13 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:18:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 3:18:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:18:13 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:18:14 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 15:18:15.065294 389 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 15:18:15 dist-test-slave-7s64 krb5kdc[11213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968695, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 15:18:15.223014 389 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 15:18:15.243976 389 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 15:18:16.034869 389 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 15:18:16.035009 389 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-kms
I20260620 15:18:16.035183 389 mini_ranger_kms.cc:227] Using postgres at 127.0.97.84:56819
2026-06-20 15:18:16,442 [I] DB FLAVOR :POSTGRES
2026-06-20 15:18:16,442 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:18:16,442 [I] Checking connection
2026-06-20 15:18:16,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 15:18:16,709 [I] connection success
2026-06-20 15:18:16,709 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 15:18:16,710 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 15:18:16,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 15:18:16,971 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 15:18:16,971 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:18:16,972 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 15:18:16,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:56819/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 15:18:17,244 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 15:18:17.717371 389 mini_ranger_kms.cc:310] Created kms service
I20260620 15:18:17.893002 389 mini_ranger_kms.cc:326] Created kudu user
I20260620 15:18:17.935644 389 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 15:18:17 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968695, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:18:18.317478 389 mini_ranger_kms.cc:384] Added ranger policy
I20260620 15:18:18.317826 389 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 15:18:18.317905 389 mini_ranger_kms.cc:240] Using host: 127.0.97.84
I20260620 15:18:18.337180 389 mini_ranger_kms.cc:276] Ranger KMS PID: 11606
I20260620 15:18:18.337373 389 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.84:54213
Jun 20, 2026 3:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:18:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 15:18:20 dist-test-slave-7s64 krb5kdc[11213](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781968700, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:18:20.625306 389 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.402s user 0.134s sys 0.292s
I20260620 15:18:20.625541 389 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 15:18:20.625730 389 mini_ranger_kms.cc:401] 127.0.97.84:54213/kms/v1/keys
Jun 20 15:18:21 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968700, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 20 15:18:21 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968700, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 20 15:18:23 dist-test-slave-7s64 krb5kdc[11213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:18:23.664022 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35357
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35357
--ranger_config_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:54213/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:18:23.995133 11657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:18:23.995394 11657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:18:23.995532 11657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:18:24.002898 11657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:18:24.002995 11657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:18:24.003064 11657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:18:24.003165 11657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:18:24.003228 11657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:18:24.012889 11657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.84:54213/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35357
--ranger_config_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:18:24.014168 11657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:18:24.015465 11657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:18:24.028884 11665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:18:24.029119 11662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:18:24.028913 11663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:18:24.029968 11657 server_base.cc:1034] running on GCE node
I20260620 15:18:24.030812 11657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:18:24.032441 11657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:18:24.033699 11657 hybrid_clock.cc:648] HybridClock initialized: now 1781968704033640 us; error 45 us; skew 500 ppm
I20260620 15:18:24.034086 11657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:18:24 dist-test-slave-7s64 krb5kdc[11213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:18:24.041090 11657 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:18:24.042644 11657 webserver.cc:460] Webserver started at http://127.0.97.126:45015/ using document root <none> and password file <none>
I20260620 15:18:24.043249 11657 fs_manager.cc:359] Metadata directory not provided
I20260620 15:18:24.043376 11657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:18:24.043658 11657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:18:24 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:18:24.103581 11657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "3cea5e63d5f24e4d90891182da1d1067"
format_stamp: "Formatted at 2026-06-20 15:18:24 on dist-test-slave-7s64"
server_key: "9f013fde73ef4713acfdbedf769e3886"
server_key_iv: "c22b9d29d722f9d502680aa58a13bb5d"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:24.104543 11657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "3cea5e63d5f24e4d90891182da1d1067"
format_stamp: "Formatted at 2026-06-20 15:18:24 on dist-test-slave-7s64"
server_key: "9f013fde73ef4713acfdbedf769e3886"
server_key_iv: "c22b9d29d722f9d502680aa58a13bb5d"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:24.109683 11657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:18:24.127813 11672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:18:24.129549 11657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:18:24.129722 11657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "3cea5e63d5f24e4d90891182da1d1067"
format_stamp: "Formatted at 2026-06-20 15:18:24 on dist-test-slave-7s64"
server_key: "9f013fde73ef4713acfdbedf769e3886"
server_key_iv: "c22b9d29d722f9d502680aa58a13bb5d"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:24.129961 11657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:18:24.154320 11657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:18:24.158912 11657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:18:24.159195 11657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:18:24.178841 11723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35357 every 8 connection(s)
I20260620 15:18:24.178838 11657 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35357
I20260620 15:18:24.180531 11657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:18:24.184681 11724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:18:24.186553 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 11657
I20260620 15:18:24.186749 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:18:24.187561 389 external_mini_cluster.cc:1428] Setting key b52b15f459c56d3986d794f55cb412ac
I20260620 15:18:24.201047 11724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067: Bootstrap starting.
I20260620 15:18:24.205945 11724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067: Neither blocks nor log segments found. Creating new log.
I20260620 15:18:24.207222 11724 log.cc:824] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067: Log is configured to *not* fsync() on all Append() calls
I20260620 15:18:24.210346 11724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067: No bootstrap required, opened a new log
Jun 20 15:18:24 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:18:24.220672 11724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cea5e63d5f24e4d90891182da1d1067" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35357 } }
I20260620 15:18:24.221168 11724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:18:24.221288 11724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cea5e63d5f24e4d90891182da1d1067, State: Initialized, Role: FOLLOWER
I20260620 15:18:24.221935 11724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cea5e63d5f24e4d90891182da1d1067" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35357 } }
I20260620 15:18:24.222219 11724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:18:24.222366 11724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:18:24.222558 11724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:18:24.223999 11727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:24.189618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.0.97.126:35357)
0620 15:18:24.204284 (+ 14666us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:24.204299 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:24.204346 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:18:24.207141 (+ 2795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:24.207165 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:24.207619 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:24.208130 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:18:24.209391 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:24.210719 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:24.211967 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:24.212686 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:24.212892 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:24.216572 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:18:24.216583 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:18:24.216596 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:18:24.216630 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:18:24.219450 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:24.220095 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:24.220110 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:24.220115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:24.220687 (+ 572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:24.221304 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:24.221323 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:24.221326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:24.221930 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:18:24.222332 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:24.222662 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 15:18:24.223051 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14446,"thread_start_us":16372,"threads_started":1}
I20260620 15:18:24.225726 11724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cea5e63d5f24e4d90891182da1d1067" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35357 } }
I20260620 15:18:24.226262 11724 leader_election.cc:302] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cea5e63d5f24e4d90891182da1d1067; no voters:
I20260620 15:18:24.226926 11724 leader_election.cc:288] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:18:24.227028 11729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:18:24.228826 11724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:18:24.234261 11729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [term 1 LEADER]: Becoming Leader. State: Replica: 3cea5e63d5f24e4d90891182da1d1067, State: Running, Role: LEADER
I20260620 15:18:24.235524 11729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cea5e63d5f24e4d90891182da1d1067" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35357 } }
I20260620 15:18:24.238546 11730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cea5e63d5f24e4d90891182da1d1067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cea5e63d5f24e4d90891182da1d1067" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35357 } } }
I20260620 15:18:24.239172 11730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [sys.catalog]: This master's current role is: LEADER
I20260620 15:18:24.239176 11731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cea5e63d5f24e4d90891182da1d1067. Latest consensus state: current_term: 1 leader_uuid: "3cea5e63d5f24e4d90891182da1d1067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cea5e63d5f24e4d90891182da1d1067" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35357 } } }
I20260620 15:18:24.239873 11731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067 [sys.catalog]: This master's current role is: LEADER
I20260620 15:18:24.246941 11740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:18:24.255920 11740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:18:24.263343 11724 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:18:24.287919 11740 catalog_manager.cc:1347] Generated new cluster ID: c572252899b047899eb93dc88e12d670
I20260620 15:18:24.288631 11740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:18:24.307384 11740 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:18:24.309456 11740 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:18:24.320863 11740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067: Generated new TSK 0
I20260620 15:18:24.322005 11740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:18:25.152143 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:54213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35357
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:18:25 dist-test-slave-7s64 krb5kdc[11213](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781968705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 15:18:25.561962 11770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:18:25.562466 11770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:18:25.562659 11770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:18:25.570130 11770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:18:25.570297 11770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:18:25.570887 11770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:18:25.582803 11770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.84:54213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:18:25.584143 11770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:18:25.585651 11770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:18:25.605109 11780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:18:25.606335 11781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:18:25.612067 11770 server_base.cc:1034] running on GCE node
W20260620 15:18:25.612434 11783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:18:25.613189 11770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:18:25.614317 11770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:18:25.615598 11770 hybrid_clock.cc:648] HybridClock initialized: now 1781968705615510 us; error 81 us; skew 500 ppm
I20260620 15:18:25.616016 11770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:18:25 dist-test-slave-7s64 krb5kdc[11213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:18:25.619939 11770 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:18:25.633813 11770 webserver.cc:460] Webserver started at http://127.0.97.65:46075/ using document root <none> and password file <none>
I20260620 15:18:25.634829 11770 fs_manager.cc:359] Metadata directory not provided
I20260620 15:18:25.634982 11770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:18:25.635406 11770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:18:25 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:18:25.663288 11770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "345fc5dd6f394446883aa2e457902af4"
format_stamp: "Formatted at 2026-06-20 15:18:25 on dist-test-slave-7s64"
server_key: "4b201bf9fbc9785e5f082f7540648703"
server_key_iv: "ffc7aab197771c73af704e2a5a878629"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:25.664487 11770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "345fc5dd6f394446883aa2e457902af4"
format_stamp: "Formatted at 2026-06-20 15:18:25 on dist-test-slave-7s64"
server_key: "4b201bf9fbc9785e5f082f7540648703"
server_key_iv: "ffc7aab197771c73af704e2a5a878629"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:25.678025 11770 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.003s sys 0.008s
I20260620 15:18:25.704097 11790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:18:25.705960 11770 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.006s
I20260620 15:18:25.706219 11770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "345fc5dd6f394446883aa2e457902af4"
format_stamp: "Formatted at 2026-06-20 15:18:25 on dist-test-slave-7s64"
server_key: "4b201bf9fbc9785e5f082f7540648703"
server_key_iv: "ffc7aab197771c73af704e2a5a878629"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:25.706545 11770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:18:25.758965 11770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:18:25.764093 11770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:18:25.764451 11770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:18:25.771277 11770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:18:25.779557 11770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:18:25.779691 11770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 15:18:25.779876 11770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:18:25.780005 11770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:18:25.847282 11902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36687 every 8 connection(s)
I20260620 15:18:25.847661 11770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36687
I20260620 15:18:25.849364 11770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:18:25.854918 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 11770
I20260620 15:18:25.855127 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:18:25.855955 389 external_mini_cluster.cc:1428] Setting key 610a31d3d1e352747522055f6a4ead29
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Jun 20 15:18:25 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260620 15:18:25.906982 11906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:25.860124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:37799 (local address 127.0.97.126:35357)
0620 15:18:25.872878 (+ 12754us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:25.872885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:25.873642 (+ 757us) server_negotiation.cc:407] Connection header received
0620 15:18:25.876958 (+ 3316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:25.876964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:25.877056 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:25.877160 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:18:25.887530 (+ 10370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:25.888843 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:25.891475 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:25.892189 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:25.892285 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:25.899430 (+ 7145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:18:25.899437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:18:25.899443 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:18:25.899482 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:18:25.903200 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:25.904155 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:25.904161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:25.904165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:25.904311 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:25.904873 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:25.904878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:25.904882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:25.905222 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:18:25.905356 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:25.906316 (+ 960us) server_negotiation.cc:299] Negotiation successful
0620 15:18:25.906552 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12613,"thread_start_us":12478,"threads_started":1}
I20260620 15:18:25.909198 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:25.859459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35357 (local address 127.0.97.65:37799)
0620 15:18:25.873465 (+ 14006us) negotiation.cc:107] Waiting for socket to connect
0620 15:18:25.873519 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:18:25.876537 (+ 3018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:18:25.877609 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:18:25.877620 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:18:25.878226 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:18:25.879002 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:25.879027 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:25.889073 (+ 10046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:25.889079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:25.891290 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:25.891316 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:25.892657 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:25.892664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:25.892971 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:25.893813 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:18:25.893862 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:18:25.899161 (+ 5299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:18:25.903397 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:18:25.903407 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:18:25.903427 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:18:25.904014 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:18:25.904456 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:18:25.904461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:18:25.904465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:18:25.904730 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:18:25.905450 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:18:25.905459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:18:25.905885 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 15:18:25.907215 (+ 1330us) client_negotiation.cc:240] Negotiation successful
0620 15:18:25.907758 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13621,"thread_start_us":13396,"threads_started":1}
I20260620 15:18:25.911321 11903 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35357
I20260620 15:18:25.911679 11903 heartbeater.cc:464] Registering TS with master...
I20260620 15:18:25.912534 11903 heartbeater.cc:511] Master 127.0.97.126:35357 requested a full tablet report, sending...
I20260620 15:18:25.915088 11689 ts_manager.cc:194] Registered new tserver with Master: 345fc5dd6f394446883aa2e457902af4 (127.0.97.65:36687)
I20260620 15:18:25.917258 11689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:37799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:18:25.931818 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:54213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35357
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:18:25 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 20 15:18:26 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
W20260620 15:18:26.371409 11909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:18:26.371922 11909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:18:26.372135 11909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:18:26.383925 11909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:18:26.384145 11909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:18:26.384392 11909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:18:26.399828 11909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.84:54213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:18:26.401361 11909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:18:26.402643 11909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:18:26.469338 11916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:18:26.470783 11917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:18:26.476751 11920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:18:26.480902 11909 server_base.cc:1034] running on GCE node
I20260620 15:18:26.481727 11909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:18:26.497713 11909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:18:26.499285 11909 hybrid_clock.cc:648] HybridClock initialized: now 1781968706499179 us; error 107 us; skew 500 ppm
I20260620 15:18:26.500253 11909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:18:26 dist-test-slave-7s64 krb5kdc[11213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:18:26.504458 11909 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:18:26.522588 11909 webserver.cc:460] Webserver started at http://127.0.97.66:37947/ using document root <none> and password file <none>
I20260620 15:18:26.523648 11909 fs_manager.cc:359] Metadata directory not provided
I20260620 15:18:26.523857 11909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:18:26.524302 11909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:18:26 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:18:26.543751 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "5bc402c124654ff19c068490d41165d6"
format_stamp: "Formatted at 2026-06-20 15:18:26 on dist-test-slave-7s64"
server_key: "853cbfae043f582670841939508c50bb"
server_key_iv: "e23a3c43174e963516117349c4c42f56"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:26.545560 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "5bc402c124654ff19c068490d41165d6"
format_stamp: "Formatted at 2026-06-20 15:18:26 on dist-test-slave-7s64"
server_key: "853cbfae043f582670841939508c50bb"
server_key_iv: "e23a3c43174e963516117349c4c42f56"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:26.566504 11909 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.019s sys 0.000s
I20260620 15:18:26.591925 11926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:18:26.602869 11909 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.006s sys 0.004s
I20260620 15:18:26.603166 11909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "5bc402c124654ff19c068490d41165d6"
format_stamp: "Formatted at 2026-06-20 15:18:26 on dist-test-slave-7s64"
server_key: "853cbfae043f582670841939508c50bb"
server_key_iv: "e23a3c43174e963516117349c4c42f56"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:26.603515 11909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:18:26.725353 11909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:18:26.730867 11909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:18:26.731166 11909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:18:26.732281 11909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:18:26.734102 11909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:18:26.734189 11909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:18:26.734349 11909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:18:26.734409 11909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:18:26.778432 11909 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:43183
I20260620 15:18:26.778474 12045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:43183 every 8 connection(s)
I20260620 15:18:26.780550 11909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:18:26.784931 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 11909
I20260620 15:18:26.785135 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:18:26.785789 389 external_mini_cluster.cc:1428] Setting key af1695842e15720c5aae33137aa67a91
Jun 20 15:18:26 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:18:26.808655 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:26.783225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:60843 (local address 127.0.97.126:35357)
0620 15:18:26.789240 (+ 6015us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:26.789246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:26.789266 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:18:26.791662 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:26.791668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:26.791760 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:26.791868 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:18:26.793812 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:26.794904 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:26.797210 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:26.797805 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:26.797935 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:26.801931 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:18:26.801937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:18:26.801942 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:18:26.801976 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:18:26.805380 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:26.806179 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:26.806184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:26.806188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:26.806320 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:26.806780 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:26.806785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:26.806788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:26.807122 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:18:26.807271 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:26.808033 (+ 762us) server_negotiation.cc:299] Negotiation successful
0620 15:18:26.808232 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5899,"thread_start_us":5782,"threads_started":1}
I20260620 15:18:26.809684 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:26.784094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35357 (local address 127.0.97.66:60843)
0620 15:18:26.786411 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0620 15:18:26.786466 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:18:26.791274 (+ 4808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:18:26.792252 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:18:26.792263 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:18:26.792872 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:18:26.793609 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:26.793632 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:26.795094 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:26.795099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:26.797049 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:26.797062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:26.798007 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:26.798014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:26.798261 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:26.799051 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:18:26.799078 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:18:26.801670 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:18:26.805527 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:18:26.805535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:18:26.805550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:18:26.806031 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:18:26.806416 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:18:26.806420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:18:26.806424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:18:26.806644 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:18:26.807235 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:18:26.807242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:18:26.807606 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 15:18:26.807897 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 15:18:26.808388 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2022,"thread_start_us":2989,"threads_started":1}
I20260620 15:18:26.811201 12046 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35357
I20260620 15:18:26.811507 12046 heartbeater.cc:464] Registering TS with master...
I20260620 15:18:26.812292 12046 heartbeater.cc:511] Master 127.0.97.126:35357 requested a full tablet report, sending...
I20260620 15:18:26.814293 11689 ts_manager.cc:194] Registered new tserver with Master: 5bc402c124654ff19c068490d41165d6 (127.0.97.66:43183)
I20260620 15:18:26.815335 11689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:60843
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:18:26.854115 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:54213/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35357
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:18:26.920890 11903 heartbeater.cc:503] Master 127.0.97.126:35357 was elected leader, sending a full tablet report...
W20260620 15:18:27.221215 12052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:18:27.221468 12052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:18:27.221553 12052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:18:27.228300 12052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:18:27.228395 12052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:18:27.228519 12052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:18:27.237978 12052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.84:54213/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:18:27.239133 12052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:18:27.240345 12052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:18:27.253463 12057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:18:27.253494 12058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:18:27.253640 12060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:18:27.257841 12052 server_base.cc:1034] running on GCE node
I20260620 15:18:27.258456 12052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:18:27.259661 12052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:18:27.260880 12052 hybrid_clock.cc:648] HybridClock initialized: now 1781968707260845 us; error 53 us; skew 500 ppm
I20260620 15:18:27.261174 12052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:18:27 dist-test-slave-7s64 krb5kdc[11213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:18:27.264030 12052 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:18:27.265388 12052 webserver.cc:460] Webserver started at http://127.0.97.67:41273/ using document root <none> and password file <none>
I20260620 15:18:27.266021 12052 fs_manager.cc:359] Metadata directory not provided
I20260620 15:18:27.266156 12052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:18:27.266440 12052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:18:27 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:18:27.282301 12052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "df823f24ed8447fcadaddc4af87721aa"
format_stamp: "Formatted at 2026-06-20 15:18:27 on dist-test-slave-7s64"
server_key: "c2ac872588c64fd6ee53d497dd29b983"
server_key_iv: "681c8e7aa58443a71b2ab8327a3c1a52"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:27.283172 12052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "df823f24ed8447fcadaddc4af87721aa"
format_stamp: "Formatted at 2026-06-20 15:18:27 on dist-test-slave-7s64"
server_key: "c2ac872588c64fd6ee53d497dd29b983"
server_key_iv: "681c8e7aa58443a71b2ab8327a3c1a52"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:27.288080 12052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:18:27.301738 12067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:18:27.303182 12052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:18:27.303342 12052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "df823f24ed8447fcadaddc4af87721aa"
format_stamp: "Formatted at 2026-06-20 15:18:27 on dist-test-slave-7s64"
server_key: "c2ac872588c64fd6ee53d497dd29b983"
server_key_iv: "681c8e7aa58443a71b2ab8327a3c1a52"
server_key_version: "kuduclusterkey@0"
I20260620 15:18:27.303525 12052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:18:27.321118 12052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:18:27.324473 12052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:18:27.324784 12052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:18:27.325667 12052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:18:27.326946 12052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:18:27.327014 12052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:18:27.327162 12052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:18:27.327220 12052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:18:27.362738 12052 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39451
I20260620 15:18:27.362766 12179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39451 every 8 connection(s)
I20260620 15:18:27.363971 12052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:18:27.374300 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 12052
I20260620 15:18:27.374531 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:18:27.375072 389 external_mini_cluster.cc:1428] Setting key e886ad0fa2ec65fcc479febdf70393a9
Jun 20 15:18:27 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:18:27.383005 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.366587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:56709 (local address 127.0.97.126:35357)
0620 15:18:27.367032 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.367036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.367845 (+ 809us) server_negotiation.cc:407] Connection header received
0620 15:18:27.369636 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.369642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.369734 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.369857 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:18:27.371175 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.371912 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.373505 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.373902 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.374034 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.378057 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:18:27.378064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:18:27.378068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:18:27.378109 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:18:27.380469 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:27.381145 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:27.381149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:27.381151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:27.381291 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:27.381666 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:27.381669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:27.381671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:27.381910 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:18:27.382017 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.382610 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.382750 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":242,"threads_started":1}
I20260620 15:18:27.383955 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.366968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35357 (local address 127.0.97.67:56709)
0620 15:18:27.367690 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 15:18:27.367725 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:18:27.369372 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:18:27.370075 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:18:27.370083 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:18:27.370481 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:18:27.371013 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.371029 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.372078 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.372090 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.373345 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.373356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.374107 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.374113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.374341 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.375056 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:18:27.375075 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:18:27.377849 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:18:27.380635 (+ 2786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:18:27.380640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:18:27.380651 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:18:27.381034 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:18:27.381381 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:18:27.381384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:18:27.381386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:18:27.381560 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:18:27.382012 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:18:27.382017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:18:27.382260 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 15:18:27.382520 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 15:18:27.382861 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":309,"threads_started":1}
I20260620 15:18:27.385296 12180 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35357
I20260620 15:18:27.385569 12180 heartbeater.cc:464] Registering TS with master...
I20260620 15:18:27.386194 12180 heartbeater.cc:511] Master 127.0.97.126:35357 requested a full tablet report, sending...
I20260620 15:18:27.387498 11689 ts_manager.cc:194] Registered new tserver with Master: df823f24ed8447fcadaddc4af87721aa (127.0.97.67:39451)
I20260620 15:18:27.388242 11689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:56709
I20260620 15:18:27.390295 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:18:27.406096 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.394549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.0.97.126:35357)
0620 15:18:27.394691 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.394694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.395218 (+ 524us) server_negotiation.cc:407] Connection header received
0620 15:18:27.395359 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.395363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.395425 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.395500 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:18:27.396752 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.397413 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.399191 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.399572 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.399658 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.401723 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:18:27.401727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:18:27.401730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:18:27.401757 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:18:27.403902 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:27.404472 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:27.404475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:27.404478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:27.404647 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:27.404983 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:27.404986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:27.404987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:27.405207 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:18:27.405366 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.405681 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.405862 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:18:27.410027 11689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:18:27.416783 11689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:18:27.449685 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.428404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39451 (local address 127.0.0.1:37710)
0620 15:18:27.429090 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 15:18:27.429135 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:18:27.429314 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:18:27.441650 (+ 12336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:18:27.441658 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:18:27.441698 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:18:27.443125 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.443141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.444985 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.444994 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.447473 (+ 2479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.447488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.448775 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.448782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.448955 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.449004 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 15:18:27.449160 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 15:18:27.449309 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":483,"spinlock_wait_cycles":896,"thread_start_us":314,"threads_started":1}
I20260620 15:18:27.450505 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.428810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.0.97.67:39451)
0620 15:18:27.440948 (+ 12138us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.440955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.440981 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:18:27.441060 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.441065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.441448 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.442105 (+ 657us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:18:27.443338 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.444867 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.447647 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.448578 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.448732 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.448944 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.450137 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.450275 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12000,"thread_start_us":11855,"threads_started":1}
I20260620 15:18:27.450788 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.429075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43183 (local address 127.0.0.1:35968)
0620 15:18:27.430064 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0620 15:18:27.430080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:18:27.430186 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:18:27.441653 (+ 11467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:18:27.441659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:18:27.441703 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:18:27.442272 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.442284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.444751 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.444755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.447524 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.447537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.449335 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.449343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.449529 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.449561 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:18:27.449622 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 15:18:27.449759 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":244,"spinlock_wait_cycles":2432,"thread_start_us":11995,"threads_started":1}
I20260620 15:18:27.451005 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.429861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.0.97.65:36687)
0620 15:18:27.430774 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.430786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.441405 (+ 10619us) server_negotiation.cc:407] Connection header received
0620 15:18:27.441682 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.441689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.441917 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.442083 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:18:27.443503 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.444560 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.448016 (+ 3456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.449438 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.449557 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.450093 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.450550 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.450701 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":496,"threads_started":1}
I20260620 15:18:27.451363 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.430074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.0.97.66:43183)
0620 15:18:27.440902 (+ 10828us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.440910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.440940 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:18:27.441048 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.441055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.441313 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.442444 (+ 1131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:18:27.443000 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.444607 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.447690 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.449116 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.450659 (+ 1543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.450842 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.450905 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.451067 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10675,"thread_start_us":11361,"threads_started":1}
I20260620 15:18:27.452603 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.429544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36687 (local address 127.0.0.1:43868)
0620 15:18:27.441287 (+ 11743us) negotiation.cc:107] Waiting for socket to connect
0620 15:18:27.441316 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:18:27.441500 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:18:27.442269 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:18:27.442274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:18:27.442318 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:18:27.443348 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.443357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.444698 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.444706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.447854 (+ 3148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.447865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.449656 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.449664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.449839 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.449870 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:18:27.449949 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 15:18:27.450070 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11637,"mutex_wait_us":161,"thread_start_us":704,"threads_started":1}
I20260620 15:18:27.454319 12115 tablet_service.cc:1467] Processing CreateTablet for tablet b4d0f1d1e18d46be90f069821442bd65 (DEFAULT_TABLE table=test-table [id=d3df29be7a9a42168bdcb885340abd46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:18:27.454734 11981 tablet_service.cc:1467] Processing CreateTablet for tablet b4d0f1d1e18d46be90f069821442bd65 (DEFAULT_TABLE table=test-table [id=d3df29be7a9a42168bdcb885340abd46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:18:27.455258 11838 tablet_service.cc:1467] Processing CreateTablet for tablet b4d0f1d1e18d46be90f069821442bd65 (DEFAULT_TABLE table=test-table [id=d3df29be7a9a42168bdcb885340abd46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:18:27.455843 12115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d0f1d1e18d46be90f069821442bd65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:18:27.455924 11981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d0f1d1e18d46be90f069821442bd65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:18:27.456398 11838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d0f1d1e18d46be90f069821442bd65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:18:27.467156 12199 tablet_bootstrap.cc:492] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: Bootstrap starting.
I20260620 15:18:27.470007 12200 tablet_bootstrap.cc:492] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: Bootstrap starting.
I20260620 15:18:27.470822 12201 tablet_bootstrap.cc:492] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: Bootstrap starting.
I20260620 15:18:27.472113 12199 tablet_bootstrap.cc:654] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: Neither blocks nor log segments found. Creating new log.
I20260620 15:18:27.473201 12200 tablet_bootstrap.cc:654] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: Neither blocks nor log segments found. Creating new log.
I20260620 15:18:27.473413 12201 tablet_bootstrap.cc:654] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: Neither blocks nor log segments found. Creating new log.
I20260620 15:18:27.473407 12199 log.cc:824] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:18:27.474699 12200 log.cc:824] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:18:27.474948 12201 log.cc:824] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: Log is configured to *not* fsync() on all Append() calls
I20260620 15:18:27.476902 12199 tablet_bootstrap.cc:492] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: No bootstrap required, opened a new log
I20260620 15:18:27.477177 12199 ts_tablet_manager.cc:1397] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260620 15:18:27.477517 12201 tablet_bootstrap.cc:492] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: No bootstrap required, opened a new log
I20260620 15:18:27.477676 12200 tablet_bootstrap.cc:492] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: No bootstrap required, opened a new log
I20260620 15:18:27.477779 12201 ts_tablet_manager.cc:1397] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 15:18:27.477924 12200 ts_tablet_manager.cc:1397] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:18:27.484870 12201 raft_consensus.cc:348] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } }
I20260620 15:18:27.485215 12201 raft_consensus.cc:374] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:18:27.485347 12201 raft_consensus.cc:729] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df823f24ed8447fcadaddc4af87721aa, State: Initialized, Role: FOLLOWER
I20260620 15:18:27.485915 12201 consensus_queue.cc:260] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } }
I20260620 15:18:27.486910 12180 heartbeater.cc:503] Master 127.0.97.126:35357 was elected leader, sending a full tablet report...
I20260620 15:18:27.487578 12201 ts_tablet_manager.cc:1428] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260620 15:18:27.487372 12199 raft_consensus.cc:348] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } }
I20260620 15:18:27.487759 12199 raft_consensus.cc:374] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:18:27.487849 12199 raft_consensus.cc:729] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345fc5dd6f394446883aa2e457902af4, State: Initialized, Role: FOLLOWER
I20260620 15:18:27.488312 12200 raft_consensus.cc:348] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } }
I20260620 15:18:27.488730 12200 raft_consensus.cc:374] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:18:27.488833 12200 raft_consensus.cc:729] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc402c124654ff19c068490d41165d6, State: Initialized, Role: FOLLOWER
I20260620 15:18:27.488854 12199 consensus_queue.cc:260] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } }
I20260620 15:18:27.489403 12200 consensus_queue.cc:260] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } }
I20260620 15:18:27.490757 12199 ts_tablet_manager.cc:1428] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 15:18:27.492709 12046 heartbeater.cc:503] Master 127.0.97.126:35357 was elected leader, sending a full tablet report...
I20260620 15:18:27.493788 12200 ts_tablet_manager.cc:1428] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260620 15:18:27.541281 12047 tablet.cc:2369] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:18:27.615571 12181 tablet.cc:2369] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:18:27.617473 11904 tablet.cc:2369] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:18:27.625988 12207 raft_consensus.cc:482] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:18:27.626264 12207 raft_consensus.cc:504] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } }
I20260620 15:18:27.627848 12207 leader_election.cc:288] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df823f24ed8447fcadaddc4af87721aa (127.0.97.67:39451), 345fc5dd6f394446883aa2e457902af4 (127.0.97.65:36687)
I20260620 15:18:27.633507 12205 raft_consensus.cc:482] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:18:27.633814 12205 raft_consensus.cc:504] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } }
I20260620 15:18:27.635143 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.628166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41145 (local address 127.0.97.67:39451)
0620 15:18:27.628346 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.628351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.628971 (+ 620us) server_negotiation.cc:407] Connection header received
0620 15:18:27.629147 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.629152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.629233 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.629329 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:18:27.630096 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.631037 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.633114 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.634129 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.634245 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.634348 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.634757 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.634899 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:18:27.635530 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.628137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39451 (local address 127.0.97.66:41145)
0620 15:18:27.628838 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 15:18:27.628858 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:18:27.628993 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:18:27.629453 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:18:27.629463 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:18:27.629487 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:18:27.629897 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.629908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.631158 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.631164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.632953 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.632964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.634414 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.634420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.634574 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.634604 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:18:27.634762 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 15:18:27.634892 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":1,"thread_start_us":372,"threads_started":1}
I20260620 15:18:27.635794 12205 leader_election.cc:288] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 345fc5dd6f394446883aa2e457902af4 (127.0.97.65:36687), 5bc402c124654ff19c068490d41165d6 (127.0.97.66:43183)
I20260620 15:18:27.636469 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.629133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:38761 (local address 127.0.97.65:36687)
0620 15:18:27.629358 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.629363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.629791 (+ 428us) server_negotiation.cc:407] Connection header received
0620 15:18:27.629903 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.629907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.629986 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.630076 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:18:27.630633 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.632150 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.634127 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.635131 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.635262 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.635400 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.636057 (+ 657us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.636204 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:18:27.638619 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.628960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36687 (local address 127.0.97.66:38761)
0620 15:18:27.629642 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 15:18:27.629658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:18:27.629780 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:18:27.630113 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:18:27.630118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:18:27.630145 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:18:27.630491 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.630502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.632312 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.632317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.633958 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.633973 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.635564 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.635568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.635835 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.635876 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:18:27.637965 (+ 2089us) client_negotiation.cc:240] Negotiation successful
0620 15:18:27.638159 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":394,"threads_started":1}
I20260620 15:18:27.638918 12135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d0f1d1e18d46be90f069821442bd65" candidate_uuid: "5bc402c124654ff19c068490d41165d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df823f24ed8447fcadaddc4af87721aa" is_pre_election: true
I20260620 15:18:27.639479 12135 raft_consensus.cc:2454] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bc402c124654ff19c068490d41165d6 in term 0.
I20260620 15:18:27.639634 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d0f1d1e18d46be90f069821442bd65" candidate_uuid: "5bc402c124654ff19c068490d41165d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345fc5dd6f394446883aa2e457902af4" is_pre_election: true
I20260620 15:18:27.640296 11858 raft_consensus.cc:2454] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bc402c124654ff19c068490d41165d6 in term 0.
I20260620 15:18:27.640336 11928 leader_election.cc:302] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bc402c124654ff19c068490d41165d6, df823f24ed8447fcadaddc4af87721aa; no voters:
I20260620 15:18:27.641927 12207 raft_consensus.cc:2781] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:18:27.642452 12207 raft_consensus.cc:482] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:18:27.642576 12207 raft_consensus.cc:3037] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:18:27.645341 12207 raft_consensus.cc:504] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } }
I20260620 15:18:27.646288 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.636178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54913 (local address 127.0.97.65:36687)
0620 15:18:27.637007 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.637015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.637035 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:18:27.637105 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.637112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.637211 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.637882 (+ 671us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:18:27.638048 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.641487 (+ 3439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.644119 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.645354 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.645480 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.645573 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.645895 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.646007 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":5413,"threads_started":1}
I20260620 15:18:27.646323 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.636165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36687 (local address 127.0.97.67:54913)
0620 15:18:27.636516 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 15:18:27.636535 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:18:27.636630 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:18:27.637397 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:18:27.637405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:18:27.637440 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:18:27.637728 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.637742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.641696 (+ 3954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.641703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.643968 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.643980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.645515 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.645525 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.645719 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.645741 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:18:27.645872 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 15:18:27.645996 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":3}
I20260620 15:18:27.646283 12135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d0f1d1e18d46be90f069821442bd65" candidate_uuid: "5bc402c124654ff19c068490d41165d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df823f24ed8447fcadaddc4af87721aa"
I20260620 15:18:27.646469 12135 raft_consensus.cc:3037] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:18:27.647090 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d0f1d1e18d46be90f069821442bd65" candidate_uuid: "df823f24ed8447fcadaddc4af87721aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345fc5dd6f394446883aa2e457902af4" is_pre_election: true
I20260620 15:18:27.647284 11858 raft_consensus.cc:2454] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df823f24ed8447fcadaddc4af87721aa in term 0.
I20260620 15:18:27.647773 12069 leader_election.cc:302] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345fc5dd6f394446883aa2e457902af4, df823f24ed8447fcadaddc4af87721aa; no voters:
I20260620 15:18:27.648273 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.636627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55831 (local address 127.0.97.66:43183)
0620 15:18:27.636822 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.636828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.641625 (+ 4797us) server_negotiation.cc:407] Connection header received
0620 15:18:27.641871 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.641877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.641964 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.642448 (+ 484us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:18:27.643168 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.644179 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.645921 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.647194 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.647300 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.647382 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.647926 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.648049 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 15:18:27.648785 12207 leader_election.cc:288] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [CANDIDATE]: Term 1 election: Requested vote from peers df823f24ed8447fcadaddc4af87721aa (127.0.97.67:39451), 345fc5dd6f394446883aa2e457902af4 (127.0.97.65:36687)
I20260620 15:18:27.648996 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.636227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43183 (local address 127.0.97.67:55831)
0620 15:18:27.641526 (+ 5299us) negotiation.cc:107] Waiting for socket to connect
0620 15:18:27.641547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:18:27.641721 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:18:27.642480 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:18:27.642485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:18:27.642509 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:18:27.642967 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.642985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.644300 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.644306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.645761 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:18:27.645773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.647560 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.647567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:18:27.647741 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.647772 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:18:27.648481 (+ 709us) client_negotiation.cc:240] Negotiation successful
0620 15:18:27.648627 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5181,"mutex_wait_us":69,"thread_start_us":5024,"threads_started":1}
I20260620 15:18:27.649202 11858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d0f1d1e18d46be90f069821442bd65" candidate_uuid: "5bc402c124654ff19c068490d41165d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "345fc5dd6f394446883aa2e457902af4"
I20260620 15:18:27.649415 11858 raft_consensus.cc:3037] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:18:27.649685 12135 raft_consensus.cc:2454] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bc402c124654ff19c068490d41165d6 in term 1.
I20260620 15:18:27.649844 12205 raft_consensus.cc:2741] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260620 15:18:27.649806 12001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d0f1d1e18d46be90f069821442bd65" candidate_uuid: "df823f24ed8447fcadaddc4af87721aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc402c124654ff19c068490d41165d6" is_pre_election: true
I20260620 15:18:27.650408 11928 leader_election.cc:302] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bc402c124654ff19c068490d41165d6, df823f24ed8447fcadaddc4af87721aa; no voters:
I20260620 15:18:27.650660 12001 raft_consensus.cc:2379] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df823f24ed8447fcadaddc4af87721aa in current term 1: Already voted for candidate 5bc402c124654ff19c068490d41165d6 in this term.
I20260620 15:18:27.650928 12207 raft_consensus.cc:2781] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:18:27.652802 12207 raft_consensus.cc:686] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 1 LEADER]: Becoming Leader. State: Replica: 5bc402c124654ff19c068490d41165d6, State: Running, Role: LEADER
I20260620 15:18:27.652912 11858 raft_consensus.cc:2454] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bc402c124654ff19c068490d41165d6 in term 1.
I20260620 15:18:27.653574 12207 consensus_queue.cc:237] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } }
I20260620 15:18:27.657575 11688 catalog_manager.cc:5495] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bc402c124654ff19c068490d41165d6 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "5bc402c124654ff19c068490d41165d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bc402c124654ff19c068490d41165d6" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43183 } health_report { overall_health: HEALTHY } } }
I20260620 15:18:27.673982 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.668902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.0.97.66:43183)
0620 15:18:27.669040 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.669043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.669088 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:18:27.669294 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.669297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.669377 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.669490 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:18:27.669901 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.670610 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.672003 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.672381 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.672469 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.673109 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:18:27.673539 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:18:27.673650 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.673736 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.673800 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 15:18:27.681674 12135 raft_consensus.cc:1261] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 1 FOLLOWER]: Refusing update from remote peer 5bc402c124654ff19c068490d41165d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:18:27.681906 11858 raft_consensus.cc:1261] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [term 1 FOLLOWER]: Refusing update from remote peer 5bc402c124654ff19c068490d41165d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:18:27.682488 12212 consensus_queue.cc:1035] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df823f24ed8447fcadaddc4af87721aa" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:18:27.682839 12207 consensus_queue.cc:1035] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "345fc5dd6f394446883aa2e457902af4" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:18:27.696447 12216 mvcc.cc:205] Tried to move back new op lower bound from 7298943826655330304 to 7298943826552578048. Current Snapshot: MvccSnapshot[applied={T|T < 7298943826655330304}]
I20260620 15:18:27.706215 11689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:18:27.706477 11689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:18:27.709872 11689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cea5e63d5f24e4d90891182da1d1067: Sending DeleteTablet for 3 replicas of tablet b4d0f1d1e18d46be90f069821442bd65
I20260620 15:18:27.710875 11981 tablet_service.cc:1514] Processing DeleteTablet for tablet b4d0f1d1e18d46be90f069821442bd65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:18:27 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:35968
I20260620 15:18:27.710914 12115 tablet_service.cc:1514] Processing DeleteTablet for tablet b4d0f1d1e18d46be90f069821442bd65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:18:27 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:37710
I20260620 15:18:27.710901 11838 tablet_service.cc:1514] Processing DeleteTablet for tablet b4d0f1d1e18d46be90f069821442bd65 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:18:27 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:43868
I20260620 15:18:27.713654 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 11770
I20260620 15:18:27.724253 12225 tablet_replica.cc:330] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: stopping tablet replica
I20260620 15:18:27.724886 12226 tablet_replica.cc:330] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: stopping tablet replica
I20260620 15:18:27.725536 12226 raft_consensus.cc:2229] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:18:27.726225 12227 tablet_replica.cc:330] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: stopping tablet replica
I20260620 15:18:27.726119 12225 raft_consensus.cc:2229] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:18:27.726639 12227 raft_consensus.cc:2229] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:18:27.727164 12227 raft_consensus.cc:2258] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:18:27.727519 12225 raft_consensus.cc:2258] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:18:27.727198 12226 raft_consensus.cc:2258] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:18:27.729378 12227 ts_tablet_manager.cc:1905] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:18:27.730854 12226 ts_tablet_manager.cc:1905] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:18:27.731696 12225 ts_tablet_manager.cc:1905] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:18:27.736588 12227 ts_tablet_manager.cc:1918] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:18:27.736733 12227 log.cc:1196] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/b4d0f1d1e18d46be90f069821442bd65
I20260620 15:18:27.737601 12227 ts_tablet_manager.cc:1939] T b4d0f1d1e18d46be90f069821442bd65 P 345fc5dd6f394446883aa2e457902af4: Deleting consensus metadata
Jun 20 15:18:27 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:18:27.739324 11674 catalog_manager.cc:4841] TS 345fc5dd6f394446883aa2e457902af4 (127.0.97.65:36687): tablet b4d0f1d1e18d46be90f069821442bd65 (table test-table [id=d3df29be7a9a42168bdcb885340abd46]) successfully deleted
I20260620 15:18:27.743799 12226 ts_tablet_manager.cc:1918] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:18:27.743950 12226 log.cc:1196] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/b4d0f1d1e18d46be90f069821442bd65
I20260620 15:18:27.744338 12225 ts_tablet_manager.cc:1918] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:18:27.744444 12226 ts_tablet_manager.cc:1939] T b4d0f1d1e18d46be90f069821442bd65 P df823f24ed8447fcadaddc4af87721aa: Deleting consensus metadata
I20260620 15:18:27.744623 12225 log.cc:1196] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/b4d0f1d1e18d46be90f069821442bd65
I20260620 15:18:27.745219 12225 ts_tablet_manager.cc:1939] T b4d0f1d1e18d46be90f069821442bd65 P 5bc402c124654ff19c068490d41165d6: Deleting consensus metadata
I20260620 15:18:27.746090 11674 catalog_manager.cc:4841] TS df823f24ed8447fcadaddc4af87721aa (127.0.97.67:39451): tablet b4d0f1d1e18d46be90f069821442bd65 (table test-table [id=d3df29be7a9a42168bdcb885340abd46]) successfully deleted
I20260620 15:18:27.746858 11676 catalog_manager.cc:4841] TS 5bc402c124654ff19c068490d41165d6 (127.0.97.66:43183): tablet b4d0f1d1e18d46be90f069821442bd65 (table test-table [id=d3df29be7a9a42168bdcb885340abd46]) successfully deleted
I20260620 15:18:27.746978 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.715330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.0.97.65:36687)
0620 15:18:27.715530 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.715539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.727418 (+ 11879us) server_negotiation.cc:407] Connection header received
0620 15:18:27.727637 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.727644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.727756 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.727865 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:18:27.729666 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.730743 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.732754 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.733407 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.733519 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.739369 (+ 5850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:18:27.739382 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:18:27.739398 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:18:27.739434 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:18:27.743618 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:27.744540 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:27.744550 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:27.744555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:27.744709 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:27.745231 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:27.745239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:27.745243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:27.745662 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:18:27.745952 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.746509 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.746727 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:18:27.748157 11898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:18:27.841778 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 11770
I20260620 15:18:27.859768 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 11909
Jun 20 15:18:27 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:18:27.876462 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:27.860602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.0.97.66:43183)
0620 15:18:27.860788 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:27.860794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:27.860814 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:18:27.860981 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:27.860988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:27.861105 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:27.861220 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:18:27.862385 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.863551 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.864951 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:27.865647 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:27.865776 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:27.870001 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:18:27.870014 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:18:27.870034 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:18:27.870075 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:18:27.873839 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:27.874460 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:27.874469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:27.874473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:27.874632 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:27.874963 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:27.874969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:27.874972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:27.875415 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:18:27.875687 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:27.875893 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 15:18:27.876118 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:18:27.877166 12041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:18:27.986317 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 11909
I20260620 15:18:28.003720 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 12052
Jun 20 15:18:28 dist-test-slave-7s64 krb5kdc[11213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:18:28.018316 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:18:28.004542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.0.97.67:39451)
0620 15:18:28.004713 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:18:28.004717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:18:28.004735 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:18:28.004843 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:18:28.004848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:18:28.004909 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:18:28.004994 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:18:28.006248 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:28.007022 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:28.008258 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:18:28.008677 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:18:28.008766 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:18:28.012907 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:18:28.012922 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:18:28.012936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:18:28.012969 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:18:28.015703 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:28.016312 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:28.016319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:28.016321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:28.016436 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:18:28.016933 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:18:28.016936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:18:28.016938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:18:28.017304 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:18:28.017508 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 15:18:28.017848 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 15:18:28.018010 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:18:28.018949 12175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:18:28.108435 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 12052
I20260620 15:18:28.125442 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 11657
I20260620 15:18:28.126636 11719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:18:28.234982 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 11657
I20260620 15:18:28.252579 389 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 15:18:28.586956 389 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 15:18:28.587134 389 mini_ranger.cc:66] Stopping Ranger...
I20260620 15:18:28.931488 389 mini_ranger.cc:68] Stopped Ranger
2026-06-20 15:18:28.931 UTC [11243] LOG: received smart shutdown request
2026-06-20 15:18:28.934 UTC [11243] LOG: background worker "logical replication launcher" (PID 11252) exited with exit code 1
2026-06-20 15:18:28.934 UTC [11247] LOG: shutting down
2026-06-20 15:18:29.077 UTC [11243] LOG: database system is shut down
2026-06-20T15:18:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48376 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:18:29 dist-test-slave-7s64 krb5kdc[12241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:18:29 dist-test-slave-7s64 krb5kdc[12241](info): set up 2 sockets
Jun 20 15:18:29 dist-test-slave-7s64 krb5kdc[12241](info): commencing operation
krb5kdc: starting...
W20260620 15:18:30.296685 389 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:18:30 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:18:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:18:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
I20260620 15:18:30.579752 389 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskRnfwZV/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/postgres -l logfile start
2026-06-20 15:18:32.946 UTC [12271] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 15:18:32.946 UTC [12271] LOG: listening on IPv4 address "127.0.97.84", port 48525
2026-06-20 15:18:32.949 UTC [12271] LOG: listening on Unix socket "/tmp/.s.PGSQL.48525"
2026-06-20 15:18:32.970 UTC [12274] LOG: database system was shut down at 2026-06-20 15:18:31 UTC
2026-06-20 15:18:32.973 UTC [12271] LOG: database system is ready to accept connections
I20260620 15:18:33.956414 389 mini_postgres.cc:96] Postgres bound to 48525
2026-06-20 15:18:33.973 UTC [12282] FATAL: database "slave" does not exist
127.0.97.84:48525 - accepting connections
I20260620 15:18:33.974340 389 mini_ranger.cc:161] Starting Ranger...
I20260620 15:18:33.998503 389 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 15:18:34.470222 389 mini_ranger.cc:87] Created ranger Postgres database
I20260620 15:18:34.470309 389 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-admin
2026-06-20 15:18:34,746 [I] DB FLAVOR :POSTGRES
2026-06-20 15:18:34,747 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:18:34,747 [I] Checking connection..
2026-06-20 15:18:34,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 15:18:34,981 [I] Checking connection passed.
2026-06-20 15:18:34,981 [I] --------- Verifying version history table ---------
2026-06-20 15:18:34,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:18:35,215 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:18:35,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:18:35,469 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:18:35,469 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 15:18:35,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 15:18:35.719 UTC [12346] WARNING: there is no transaction in progress
2026-06-20 15:18:35,726 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 15:18:35,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:18:35,960 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 15:18:35,961 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:18:35,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 15:18:36,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 15:18:36,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7s64.c.gcp-upstream.internal','N') ;"
2026-06-20 15:18:36,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:18:36,900 [I] Table x_portal_user does not exist in database ranger
2026-06-20 15:18:36,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:18:37,149 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 15:18:37,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:18:37,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:18:37,629 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 15:18:37,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 15:18:38.536 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.551 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.566 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.580 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.599 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.619 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.683 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.689 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.702 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.712 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.722 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.734 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.743 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.750 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.757 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:38.763 UTC [12454] WARNING: there is no transaction in progress
2026-06-20 15:18:39,389 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 15:18:39,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 15:18:39,604 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 15:18:39,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:18:39,831 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 15:18:39,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:18:40,067 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 15:18:40,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:18:40,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:18:40,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7s64.c.gcp-upstream.internal';"
2026-06-20 15:18:40,760 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 15:18:40,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 15:18:41,012 [I] DB_PATCHES have already been applied
I20260620 15:18:41.018427 389 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-admin:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskRnfwZV/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 15:18:41.018512 389 mini_ranger.cc:193] Using host: 127.0.97.84
I20260620 15:18:41.032449 389 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.84:45571
Jun 20, 2026 3:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:18:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 15:18:42.165406 389 mini_ranger.cc:160] Time spent starting Ranger: real 8.191s user 0.000s sys 0.034s
Jun 20, 2026 3:19:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:19:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 3:19:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:19:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:19:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 15:19:02.979339 389 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 15:19:03 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968743, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 15:19:03.142714 389 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 15:19:03.162626 389 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 15:19:03.927227 389 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 15:19:03.927318 389 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-kms
I20260620 15:19:03.927353 389 mini_ranger_kms.cc:227] Using postgres at 127.0.97.84:48525
2026-06-20 15:19:04,269 [I] DB FLAVOR :POSTGRES
2026-06-20 15:19:04,269 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:19:04,269 [I] Checking connection
2026-06-20 15:19:04,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 15:19:04,506 [I] connection success
2026-06-20 15:19:04,506 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 15:19:04,506 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 15:19:04,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 15:19:04,730 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 15:19:04,731 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:19:04,731 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 15:19:04,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:48525/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 15:19:04,989 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 15:19:05.409917 389 mini_ranger_kms.cc:310] Created kms service
I20260620 15:19:05.609858 389 mini_ranger_kms.cc:326] Created kudu user
I20260620 15:19:05.652624 389 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 15:19:05 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968743, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:19:06.028999 389 mini_ranger_kms.cc:384] Added ranger policy
I20260620 15:19:06.029318 389 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-kms:/tmp/dist-test-taskRnfwZV/build/asan/bin/postgresql.jar:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskRnfwZV/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 15:19:06.029377 389 mini_ranger_kms.cc:240] Using host: 127.0.97.84
I20260620 15:19:06.046365 389 mini_ranger_kms.cc:276] Ranger KMS PID: 12634
I20260620 15:19:06.046499 389 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.84:46103
Jun 20, 2026 3:19:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:19:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:19:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:19:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:19:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:19:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 15:19:07 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781968747, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:08.309226 389 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.167s user 0.066s sys 0.291s
I20260620 15:19:08.309487 389 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 15:19:08.309649 389 mini_ranger_kms.cc:401] 127.0.97.84:46103/kms/v1/keys
Jun 20 15:19:08 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968747, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 20 15:19:08 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968747, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 20 15:19:11 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:19:11.195865 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37675
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37675
--ranger_config_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:46103/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:11.507295 12685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:11.507541 12685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:11.507618 12685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:11.514485 12685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:19:11.514571 12685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:11.514616 12685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:11.514669 12685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:19:11.514714 12685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:19:11.523911 12685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.84:46103/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37675
--ranger_config_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:11.525025 12685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:11.526237 12685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:11.537647 12690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:11.537703 12691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:11.537604 12693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:11.537926 12685 server_base.cc:1034] running on GCE node
I20260620 15:19:11.538846 12685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:11.540131 12685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:11.541360 12685 hybrid_clock.cc:648] HybridClock initialized: now 1781968751541314 us; error 49 us; skew 500 ppm
I20260620 15:19:11.541702 12685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:11 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:11.544705 12685 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:19:11.545984 12685 webserver.cc:460] Webserver started at http://127.0.97.126:41375/ using document root <none> and password file <none>
I20260620 15:19:11.546561 12685 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:11.546654 12685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:11.546890 12685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:19:11 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:19:11.606138 12685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "b39801a8a25e4d8392d586dc8484bcad"
format_stamp: "Formatted at 2026-06-20 15:19:11 on dist-test-slave-7s64"
server_key: "a0c2b54c941544b67e413ad48b27353c"
server_key_iv: "cd9ed00b653e26b60417fc6ea1a328ba"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:11.606977 12685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "b39801a8a25e4d8392d586dc8484bcad"
format_stamp: "Formatted at 2026-06-20 15:19:11 on dist-test-slave-7s64"
server_key: "a0c2b54c941544b67e413ad48b27353c"
server_key_iv: "cd9ed00b653e26b60417fc6ea1a328ba"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:11.612145 12685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:19:11.627135 12700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:11.628723 12685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:19:11.628899 12685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "b39801a8a25e4d8392d586dc8484bcad"
format_stamp: "Formatted at 2026-06-20 15:19:11 on dist-test-slave-7s64"
server_key: "a0c2b54c941544b67e413ad48b27353c"
server_key_iv: "cd9ed00b653e26b60417fc6ea1a328ba"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:11.629118 12685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:11.648380 12685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:11.652333 12685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:11.652667 12685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:11.671079 12685 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37675
I20260620 15:19:11.671123 12751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37675 every 8 connection(s)
I20260620 15:19:11.672452 12685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:19:11.674793 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 12685
I20260620 15:19:11.675019 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:19:11.675856 389 external_mini_cluster.cc:1428] Setting key 8ae89f66be3f6e9c546b10fea10d1f16
I20260620 15:19:11.676894 12752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:19:11 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:11.690790 12752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: Bootstrap starting.
I20260620 15:19:11.693885 12752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:11.694880 12752 log.cc:824] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:11.697535 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:11.677886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.0.97.126:37675)
0620 15:19:11.678615 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:11.678626 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:11.678665 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:19:11.680111 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:11.680130 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:11.680579 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:11.680905 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:11.682449 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:11.683759 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:11.685577 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:11.686215 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:11.686346 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:11.690703 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:11.690711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:11.690725 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:11.690762 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:11.693706 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:11.694457 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:11.694465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:11.694476 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:11.694627 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:11.694993 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:11.694999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:11.695002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:11.695480 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:11.695809 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:11.696246 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 15:19:11.696629 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":361,"threads_started":1}
I20260620 15:19:11.697810 12752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: No bootstrap required, opened a new log
I20260620 15:19:11.705468 12752 raft_consensus.cc:348] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } }
I20260620 15:19:11.705813 12752 raft_consensus.cc:374] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:11.705928 12752 raft_consensus.cc:729] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39801a8a25e4d8392d586dc8484bcad, State: Initialized, Role: FOLLOWER
I20260620 15:19:11.706450 12752 consensus_queue.cc:260] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } }
I20260620 15:19:11.706724 12752 raft_consensus.cc:388] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:19:11.706864 12752 raft_consensus.cc:482] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:19:11.707036 12752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:11.709474 12752 raft_consensus.cc:504] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } }
I20260620 15:19:11.709950 12752 leader_election.cc:302] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b39801a8a25e4d8392d586dc8484bcad; no voters:
I20260620 15:19:11.710489 12752 leader_election.cc:288] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:19:11.710614 12757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:19:11.711287 12757 raft_consensus.cc:686] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 1 LEADER]: Becoming Leader. State: Replica: b39801a8a25e4d8392d586dc8484bcad, State: Running, Role: LEADER
I20260620 15:19:11.711802 12757 consensus_queue.cc:237] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } }
I20260620 15:19:11.712481 12752 sys_catalog.cc:564] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:19:11.714504 12758 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b39801a8a25e4d8392d586dc8484bcad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } } }
I20260620 15:19:11.714911 12758 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:11.715332 12759 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [sys.catalog]: SysCatalogTable state changed. Reason: New leader b39801a8a25e4d8392d586dc8484bcad. Latest consensus state: current_term: 1 leader_uuid: "b39801a8a25e4d8392d586dc8484bcad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } } }
I20260620 15:19:11.715619 12759 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:11.717819 12765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:19:11.723167 12765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:19:11.723865 12752 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:19:11.741544 12765 catalog_manager.cc:1347] Generated new cluster ID: c987f3e7bb6740f5b1568239adf25875
I20260620 15:19:11.741693 12765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:19:11.756789 12765 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:19:11.758821 12765 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:19:11.776010 12765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: Generated new TSK 0
I20260620 15:19:11.776960 12765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:19:12.477465 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:46103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37675
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:19:12 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781968752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 15:19:12.861706 12797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:12.862053 12797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:12.862259 12797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:12.873517 12797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:12.873669 12797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:12.873867 12797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:19:12.889061 12797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.84:46103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:12.890674 12797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:12.892428 12797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:12.908268 12807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:12.909781 12808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:12.917210 12810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:12.917603 12797 server_base.cc:1034] running on GCE node
I20260620 15:19:12.918320 12797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:12.919528 12797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:12.920845 12797 hybrid_clock.cc:648] HybridClock initialized: now 1781968752920739 us; error 92 us; skew 500 ppm
I20260620 15:19:12.921275 12797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:12 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:12.925417 12797 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:19:12.930955 12797 webserver.cc:460] Webserver started at http://127.0.97.65:40449/ using document root <none> and password file <none>
I20260620 15:19:12.931818 12797 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:12.932008 12797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:12.932385 12797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:19:12 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:19:12.959339 12797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "7b2c0307fa5942c2b5b0d02368a60563"
format_stamp: "Formatted at 2026-06-20 15:19:12 on dist-test-slave-7s64"
server_key: "9b14c5b42d50604633df0866b2069007"
server_key_iv: "2da5dbeb7c1c408d0151620ec927ac91"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:12.960520 12797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "7b2c0307fa5942c2b5b0d02368a60563"
format_stamp: "Formatted at 2026-06-20 15:19:12 on dist-test-slave-7s64"
server_key: "9b14c5b42d50604633df0866b2069007"
server_key_iv: "2da5dbeb7c1c408d0151620ec927ac91"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:12.967726 12797 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 15:19:13.001197 12817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:13.005980 12797 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.005s sys 0.011s
I20260620 15:19:13.006261 12797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "7b2c0307fa5942c2b5b0d02368a60563"
format_stamp: "Formatted at 2026-06-20 15:19:12 on dist-test-slave-7s64"
server_key: "9b14c5b42d50604633df0866b2069007"
server_key_iv: "2da5dbeb7c1c408d0151620ec927ac91"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:13.006728 12797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:13.089458 12797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:13.094971 12797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:13.095490 12797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:13.097035 12797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:13.098824 12797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:13.098979 12797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 15:19:13.099184 12797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:13.099295 12797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:13.184587 12797 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36451
I20260620 15:19:13.184648 12929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36451 every 8 connection(s)
I20260620 15:19:13.186286 12797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:19:13.196094 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 12797
I20260620 15:19:13.196478 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:19:13.197381 389 external_mini_cluster.cc:1428] Setting key b13eef9e077a4a6c19f5224c982cba2d
Jun 20 15:19:13 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:13.224262 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:13.189913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37675 (local address 127.0.97.65:57347)
0620 15:19:13.197040 (+ 7127us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:13.197099 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:13.199545 (+ 2446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:13.205547 (+ 6002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:13.205560 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:13.206181 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:13.206987 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:13.207027 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:13.208569 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:13.208574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:13.210579 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:13.210624 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:13.211671 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:13.211679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:13.211943 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:13.212749 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:13.212793 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:13.215569 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:13.219617 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:13.219625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:13.219640 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:13.220216 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:13.220685 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:13.220691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:13.220696 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:13.220947 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:13.221665 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:13.221673 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:13.222064 (+ 391us) client_negotiation.cc:769] Sending connection context
0620 15:19:13.222372 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 15:19:13.222866 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6795,"thread_start_us":8211,"threads_started":1}
I20260620 15:19:13.225396 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:13.190567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57347 (local address 127.0.97.126:37675)
0620 15:19:13.204752 (+ 14185us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:13.204758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:13.204779 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:19:13.204894 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:13.204900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:13.205004 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:13.205125 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:13.207217 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:13.208358 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:13.210770 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:13.211423 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:13.211559 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:13.215819 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:13.215827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:13.215832 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:13.215870 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:13.219454 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:13.220369 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:13.220376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:13.220379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:13.220547 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:13.221100 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:13.221105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:13.221108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:13.221447 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:13.221580 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:13.224790 (+ 3210us) server_negotiation.cc:299] Negotiation successful
0620 15:19:13.225010 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14051,"thread_start_us":8703,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:19:13.227552 12930 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37675
I20260620 15:19:13.227880 12930 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:13.228713 12930 heartbeater.cc:511] Master 127.0.97.126:37675 requested a full tablet report, sending...
I20260620 15:19:13.231161 12717 ts_manager.cc:194] Registered new tserver with Master: 7b2c0307fa5942c2b5b0d02368a60563 (127.0.97.65:36451)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260620 15:19:13.233393 12717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:57347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:19:13.263859 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:46103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37675
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:19:13 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 20 15:19:13 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
W20260620 15:19:13.666919 12936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:13.667341 12936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:13.667539 12936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:13.677940 12936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:13.678078 12936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:13.678287 12936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:19:13.691802 12936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.84:46103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:13.693578 12936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:13.695055 12936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:13.729451 12943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:13.729480 12944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:13.739266 12936 server_base.cc:1034] running on GCE node
W20260620 15:19:13.739753 12947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:13.740658 12936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:13.744992 12936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:13.746281 12936 hybrid_clock.cc:648] HybridClock initialized: now 1781968753746194 us; error 72 us; skew 500 ppm
I20260620 15:19:13.746731 12936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:13 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:13.755725 12936 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:19:13.766150 12936 webserver.cc:460] Webserver started at http://127.0.97.66:40431/ using document root <none> and password file <none>
I20260620 15:19:13.767128 12936 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:13.767267 12936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:13.767619 12936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:19:13 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:19:13.789816 12936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "af7efe03f44049b4ace3ba3d67b22493"
format_stamp: "Formatted at 2026-06-20 15:19:13 on dist-test-slave-7s64"
server_key: "b194638a63723febe99769e5a99daed0"
server_key_iv: "d62aa0673365c2d5d47596df49c7e5cc"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:13.790906 12936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "af7efe03f44049b4ace3ba3d67b22493"
format_stamp: "Formatted at 2026-06-20 15:19:13 on dist-test-slave-7s64"
server_key: "b194638a63723febe99769e5a99daed0"
server_key_iv: "d62aa0673365c2d5d47596df49c7e5cc"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:13.799214 12936 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260620 15:19:13.849211 12953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:13.851236 12936 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.005s
I20260620 15:19:13.851537 12936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "af7efe03f44049b4ace3ba3d67b22493"
format_stamp: "Formatted at 2026-06-20 15:19:13 on dist-test-slave-7s64"
server_key: "b194638a63723febe99769e5a99daed0"
server_key_iv: "d62aa0673365c2d5d47596df49c7e5cc"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:13.851877 12936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:13.915089 12936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:13.920120 12936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:13.920431 12936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:13.921607 12936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:13.923249 12936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:13.923337 12936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:13.923496 12936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:13.923560 12936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:13.966820 12936 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:34847
I20260620 15:19:13.966837 13072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:34847 every 8 connection(s)
I20260620 15:19:13.968070 12936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:19:13.976867 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 12936
I20260620 15:19:13.977049 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:19:13.977617 389 external_mini_cluster.cc:1428] Setting key 9bbe49a0495815c1c3bd43cf83b784fa
Jun 20 15:19:13 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:13.991453 13075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:13.970716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:33745 (local address 127.0.97.126:37675)
0620 15:19:13.971203 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:13.971208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:13.972194 (+ 986us) server_negotiation.cc:407] Connection header received
0620 15:19:13.974011 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:13.974016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:13.974119 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:13.974261 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:13.975577 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:13.976319 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:13.977956 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:13.978472 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:13.978610 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:13.982633 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:13.982642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:13.982649 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:13.982696 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:13.986470 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:13.988596 (+ 2126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:13.988604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:13.988611 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:13.988755 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:13.989280 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:13.989287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:13.989294 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:13.989678 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:13.989822 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:13.990702 (+ 880us) server_negotiation.cc:299] Negotiation successful
0620 15:19:13.990953 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
I20260620 15:19:13.992317 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:13.971306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37675 (local address 127.0.97.66:33745)
0620 15:19:13.972047 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:13.972087 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:13.973751 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:13.974476 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:13.974484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:13.974890 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:13.975417 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:13.975432 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:13.976467 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:13.976477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:13.977813 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:13.977824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:13.978692 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:13.978701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:13.978942 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:13.979668 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:13.979704 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:13.982398 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:13.987871 (+ 5473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:13.987880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:13.987896 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:13.988455 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:13.988884 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:13.988888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:13.988892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:13.989155 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:13.989824 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:13.989835 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:13.990253 (+ 418us) client_negotiation.cc:769] Sending connection context
0620 15:19:13.990546 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 15:19:13.991036 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":317,"threads_started":1}
I20260620 15:19:13.993901 13073 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37675
I20260620 15:19:13.994202 13073 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:13.994987 13073 heartbeater.cc:511] Master 127.0.97.126:37675 requested a full tablet report, sending...
I20260620 15:19:13.996572 12717 ts_manager.cc:194] Registered new tserver with Master: af7efe03f44049b4ace3ba3d67b22493 (127.0.97.66:34847)
I20260620 15:19:13.997639 12717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:33745
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:19:14.028738 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:46103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37675
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:19:14.237084 12930 heartbeater.cc:503] Master 127.0.97.126:37675 was elected leader, sending a full tablet report...
W20260620 15:19:14.345000 13079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:14.345284 13079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:14.345433 13079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:14.352346 13079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:14.352517 13079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:14.352686 13079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:19:14.362205 13079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.84:46103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:14.363346 13079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:14.364586 13079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:14.376875 13084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:14.376916 13085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:14.377408 13087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:14.377833 13079 server_base.cc:1034] running on GCE node
I20260620 15:19:14.378304 13079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:14.379055 13079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:14.380239 13079 hybrid_clock.cc:648] HybridClock initialized: now 1781968754380200 us; error 39 us; skew 500 ppm
I20260620 15:19:14.380576 13079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:14 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:14.383442 13079 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:19:14.384888 13079 webserver.cc:460] Webserver started at http://127.0.97.67:37903/ using document root <none> and password file <none>
I20260620 15:19:14.385509 13079 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:14.385645 13079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:14.385921 13079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:19:14 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:19:14.399958 13079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "cbb6cb83e03740f19bbf7686a18ccafb"
format_stamp: "Formatted at 2026-06-20 15:19:14 on dist-test-slave-7s64"
server_key: "0db24d2cea8b1c3ff7d5302472433ee9"
server_key_iv: "2de0fa0d725c59f8c27d5ca11c4e9bbf"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:14.400866 13079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "cbb6cb83e03740f19bbf7686a18ccafb"
format_stamp: "Formatted at 2026-06-20 15:19:14 on dist-test-slave-7s64"
server_key: "0db24d2cea8b1c3ff7d5302472433ee9"
server_key_iv: "2de0fa0d725c59f8c27d5ca11c4e9bbf"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:14.406102 13079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:19:14.420008 13094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:14.421319 13079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:19:14.421483 13079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "cbb6cb83e03740f19bbf7686a18ccafb"
format_stamp: "Formatted at 2026-06-20 15:19:14 on dist-test-slave-7s64"
server_key: "0db24d2cea8b1c3ff7d5302472433ee9"
server_key_iv: "2de0fa0d725c59f8c27d5ca11c4e9bbf"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:14.421685 13079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:14.442260 13079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:14.445958 13079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:14.446231 13079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:14.447058 13079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:14.448272 13079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:14.448366 13079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:14.448527 13079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:14.448606 13079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 15:19:14.479887 13079 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:40503
I20260620 15:19:14.479933 13206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:40503 every 8 connection(s)
I20260620 15:19:14.481078 13079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:19:14.489208 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 13079
I20260620 15:19:14.489383 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:19:14.489962 389 external_mini_cluster.cc:1428] Setting key 27986706c0a13615ddff1a0e586914c3
Jun 20 15:19:14 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:14.500160 13075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:14.483415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:53033 (local address 127.0.97.126:37675)
0620 15:19:14.483633 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:14.483639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:14.484812 (+ 1173us) server_negotiation.cc:407] Connection header received
0620 15:19:14.486404 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:14.486408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:14.486513 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:14.486602 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:14.487832 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:14.488596 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:14.490720 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:14.491154 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:14.491290 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:14.495063 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:14.495068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:14.495071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:14.495101 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:14.497594 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:14.498238 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:14.498242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:14.498244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:14.498370 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:14.498720 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:14.498723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:14.498725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:14.498981 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:14.499086 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:14.499740 (+ 654us) server_negotiation.cc:299] Negotiation successful
0620 15:19:14.499916 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:19:14.501084 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:14.483892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37675 (local address 127.0.97.67:53033)
0620 15:19:14.484670 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:14.484711 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:14.486172 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:14.486794 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:14.486801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:14.487179 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:14.487701 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:14.487717 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:14.488708 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:14.488711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:14.490554 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:14.490572 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:14.491386 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:14.491395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:14.491815 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:14.492597 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:14.492624 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:14.494863 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:14.497731 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:14.497737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:14.497747 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:14.498139 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:14.498463 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:14.498466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:14.498469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:14.498632 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:14.499124 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:14.499129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:14.499408 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 15:19:14.499635 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 15:19:14.499975 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":309,"threads_started":1}
I20260620 15:19:14.502347 13207 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37675
I20260620 15:19:14.502611 13207 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:14.503257 13207 heartbeater.cc:511] Master 127.0.97.126:37675 requested a full tablet report, sending...
I20260620 15:19:14.504849 12717 ts_manager.cc:194] Registered new tserver with Master: cbb6cb83e03740f19bbf7686a18ccafb (127.0.97.67:40503)
I20260620 15:19:14.505694 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:19:14.505832 12717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:53033
I20260620 15:19:14.508015 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 12797
Jun 20 15:19:14 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:19:14.524478 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:14.508910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.0.97.65:36451)
0620 15:19:14.509514 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:14.509523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:14.509551 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:19:14.509675 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:14.509682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:14.509856 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:14.510009 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:14.511590 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:14.512744 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:14.514317 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:14.514828 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:14.514963 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:14.518978 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:14.518991 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:14.519004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:14.519039 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:14.521638 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:14.522338 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:14.522351 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:14.522354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:14.522497 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:14.522875 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:14.522881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:14.522884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:14.523208 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:14.523504 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:14.523794 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 15:19:14.524072 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":278,"threads_started":1}
I20260620 15:19:14.525817 12925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:14.607914 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 12797
I20260620 15:19:14.623606 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 12936
Jun 20 15:19:14 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:19:14.639353 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:14.624532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.0.97.66:34847)
0620 15:19:14.625059 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:14.625065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:14.625089 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:19:14.625145 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:14.625159 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:14.625429 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:14.625581 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:14.626959 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:14.627940 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:14.629381 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:14.629908 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:14.630037 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:14.634161 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:14.634170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:14.634183 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:14.634239 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:14.637090 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:14.637667 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:14.637674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:14.637676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:14.637800 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:14.638100 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:14.638103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:14.638105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:14.638402 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:14.638639 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:14.638899 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 15:19:14.639086 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":272,"threads_started":1}
I20260620 15:19:14.640569 13068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:14.727316 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 12936
I20260620 15:19:14.742858 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 13079
Jun 20 15:19:14 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:19:14.758358 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:14.743886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.0.97.67:40503)
0620 15:19:14.744565 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:14.744572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:14.744601 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:19:14.744660 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:14.744666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:14.744883 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:14.745048 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:14.746299 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:14.747286 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:14.748673 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:14.749200 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:14.749295 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:14.753330 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:14.753340 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:14.753352 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:14.753380 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:14.756083 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:14.756705 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:14.756711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:14.756713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:14.756824 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:14.757116 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:14.757119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:14.757121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:14.757398 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:14.757630 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:14.757831 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 15:19:14.758038 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":325,"threads_started":1}
I20260620 15:19:14.759541 13202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:14.840050 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 13079
I20260620 15:19:14.855410 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 12685
I20260620 15:19:14.856772 12747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:14.959549 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 12685
I20260620 15:19:14.981089 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37675
--webserver_interface=127.0.97.126
--webserver_port=41375
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37675
--ranger_config_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:46103/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:46103/kms with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:15.295662 13218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:15.295900 13218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:15.295982 13218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:15.302968 13218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:19:15.303054 13218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:15.303097 13218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:15.303150 13218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:19:15.303195 13218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:19:15.312577 13218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:46103/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37675
--ranger_config_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=41375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:15.313596 13218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:15.314738 13218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:15.325928 13223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:15.325968 13224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:15.326020 13226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:15.326679 13218 server_base.cc:1034] running on GCE node
I20260620 15:19:15.327426 13218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:15.328866 13218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:15.330096 13218 hybrid_clock.cc:648] HybridClock initialized: now 1781968755330066 us; error 40 us; skew 500 ppm
I20260620 15:19:15.330443 13218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:15 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:15.333844 13218 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:19:15.335412 13218 webserver.cc:460] Webserver started at http://127.0.97.126:41375/ using document root <none> and password file <none>
I20260620 15:19:15.336300 13218 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:15.336445 13218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:19:15 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:19:15.390123 13218 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 15:19:15.393316 13234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:15.394753 13218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:19:15.394934 13218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "b39801a8a25e4d8392d586dc8484bcad"
format_stamp: "Formatted at 2026-06-20 15:19:11 on dist-test-slave-7s64"
server_key: "a0c2b54c941544b67e413ad48b27353c"
server_key_iv: "cd9ed00b653e26b60417fc6ea1a328ba"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:15.395885 13218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:15.438544 13218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:15.442348 13218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:15.442624 13218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:15.460031 13218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37675
I20260620 15:19:15.460057 13285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37675 every 8 connection(s)
I20260620 15:19:15.461455 13218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:19:15.467979 13286 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:19:15.471084 13286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: Bootstrap starting.
I20260620 15:19:15.471866 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 13218
I20260620 15:19:15.472754 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:46103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:36451
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:37675
--webserver_port=40449
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:46103/kms with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:19:15.489017 13286 log.cc:824] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:15.497421 13286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:19:15.497927 13286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: Bootstrap complete.
I20260620 15:19:15.506878 13286 raft_consensus.cc:348] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } }
I20260620 15:19:15.507714 13286 raft_consensus.cc:729] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b39801a8a25e4d8392d586dc8484bcad, State: Initialized, Role: FOLLOWER
I20260620 15:19:15.508311 13286 consensus_queue.cc:260] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } }
I20260620 15:19:15.508565 13286 raft_consensus.cc:388] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:19:15.508680 13286 raft_consensus.cc:482] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:19:15.508852 13286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:19:15.512324 13286 raft_consensus.cc:504] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } }
I20260620 15:19:15.512791 13286 leader_election.cc:302] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b39801a8a25e4d8392d586dc8484bcad; no voters:
I20260620 15:19:15.513366 13286 leader_election.cc:288] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:19:15.513506 13290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:19:15.514690 13290 raft_consensus.cc:686] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [term 2 LEADER]: Becoming Leader. State: Replica: b39801a8a25e4d8392d586dc8484bcad, State: Running, Role: LEADER
I20260620 15:19:15.515245 13290 consensus_queue.cc:237] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } }
I20260620 15:19:15.515468 13286 sys_catalog.cc:564] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:19:15.517678 13292 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [sys.catalog]: SysCatalogTable state changed. Reason: New leader b39801a8a25e4d8392d586dc8484bcad. Latest consensus state: current_term: 2 leader_uuid: "b39801a8a25e4d8392d586dc8484bcad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } } }
I20260620 15:19:15.518023 13292 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:15.518527 13291 sys_catalog.cc:455] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b39801a8a25e4d8392d586dc8484bcad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b39801a8a25e4d8392d586dc8484bcad" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37675 } } }
I20260620 15:19:15.518812 13291 sys_catalog.cc:458] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:15.529606 13296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:19:15.535043 13296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:19:15.536379 13296 catalog_manager.cc:1259] Loaded cluster ID: c987f3e7bb6740f5b1568239adf25875
I20260620 15:19:15.536562 13296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:19:15.540385 13296 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:19:15.541865 13296 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: Loaded TSK: 0
I20260620 15:19:15.543344 13296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:19:15.546684 13286 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 15:19:15.881001 13288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:15.881470 13288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:15.881695 13288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:15.893257 13288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:15.893451 13288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:15.893675 13288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:19:15.909322 13288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:46103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:36451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=40449
--tserver_master_addrs=127.0.97.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:15.911139 13288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:15.912807 13288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:15.941504 13323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:15.941653 13324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:15.949438 13288 server_base.cc:1034] running on GCE node
W20260620 15:19:15.949819 13327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:15.950489 13288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:15.951861 13288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:15.955247 13288 hybrid_clock.cc:648] HybridClock initialized: now 1781968755955170 us; error 52 us; skew 500 ppm
I20260620 15:19:15.955585 13288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:15 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:15.959693 13288 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:19:15.980039 13288 webserver.cc:460] Webserver started at http://127.0.97.65:40449/ using document root <none> and password file <none>
I20260620 15:19:15.980952 13288 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:15.981115 13288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:19:15 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:19:16.003175 13288 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:19:16.021168 13334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:16.023227 13288 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.005s sys 0.007s
I20260620 15:19:16.023504 13288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "7b2c0307fa5942c2b5b0d02368a60563"
format_stamp: "Formatted at 2026-06-20 15:19:12 on dist-test-slave-7s64"
server_key: "9b14c5b42d50604633df0866b2069007"
server_key_iv: "2da5dbeb7c1c408d0151620ec927ac91"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:16.025100 13288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:16.092726 13288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:16.098620 13288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:16.099051 13288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:16.110540 13288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:16.122817 13288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:16.123013 13288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:16.123240 13288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:16.123405 13288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:16.177191 13288 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36451
I20260620 15:19:16.177250 13447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36451 every 8 connection(s)
I20260620 15:19:16.179178 13288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:19:16.189371 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 13288
I20260620 15:19:16.190151 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:46103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:34847
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:37675
--webserver_port=40431
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:46103/kms with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:19:16 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:16.210708 13451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:16.182338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37675 (local address 127.0.97.65:37021)
0620 15:19:16.183266 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:16.183319 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:16.185674 (+ 2355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:16.187107 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:16.187118 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:16.187661 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:16.188299 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:16.188316 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:16.191271 (+ 2955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:16.191276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:16.194483 (+ 3207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:16.194498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:16.196542 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:16.196552 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:16.196812 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:16.200234 (+ 3422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:16.200265 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:16.202762 (+ 2497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:16.206241 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:16.206249 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:16.206263 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:16.206800 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:16.207323 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:16.207328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:16.207331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:16.207523 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:16.208215 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:16.208223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:16.208594 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 15:19:16.208890 (+ 296us) client_negotiation.cc:240] Negotiation successful
0620 15:19:16.209409 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":503,"threads_started":1}
I20260620 15:19:16.211688 13450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:16.182202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:37021 (local address 127.0.97.126:37675)
0620 15:19:16.182959 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:16.182978 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:16.183477 (+ 499us) server_negotiation.cc:407] Connection header received
0620 15:19:16.186254 (+ 2777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:16.186267 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:16.186583 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:16.186973 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:16.188537 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:16.191029 (+ 2492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:16.194694 (+ 3665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:16.195713 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:16.195884 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:16.203016 (+ 7132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:16.203023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:16.203036 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:16.203072 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:16.206055 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:16.206975 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:16.206984 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:16.206991 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:16.207181 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:16.207643 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:16.207648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:16.207651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:16.208080 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:16.208277 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:16.210013 (+ 1736us) server_negotiation.cc:299] Negotiation successful
0620 15:19:16.210369 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":310,"threads_started":1}
I20260620 15:19:16.214775 13448 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37675
I20260620 15:19:16.215073 13448 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:16.215740 13448 heartbeater.cc:511] Master 127.0.97.126:37675 requested a full tablet report, sending...
W20260620 15:19:16.218561 13448 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:37675 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:19:16.218765 13448 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:16.218991 13448 heartbeater.cc:511] Master 127.0.97.126:37675 requested a full tablet report, sending...
I20260620 15:19:16.220139 13448 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:16.220403 13448 heartbeater.cc:511] Master 127.0.97.126:37675 requested a full tablet report, sending...
W20260620 15:19:16.221419 13448 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:19:16.595268 13452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:16.595672 13452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:16.595852 13452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:16.603736 13452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:16.603869 13452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:16.604051 13452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:19:16.613680 13452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:46103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:34847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=40431
--tserver_master_addrs=127.0.97.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:16.614854 13452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:16.616298 13452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:16.644971 13466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:16.645139 13467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:16.649171 13469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:16.650056 13452 server_base.cc:1034] running on GCE node
I20260620 15:19:16.650830 13452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:16.651827 13452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:16.653083 13452 hybrid_clock.cc:648] HybridClock initialized: now 1781968756653007 us; error 54 us; skew 500 ppm
I20260620 15:19:16.653389 13452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:16 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:16.656437 13452 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:19:16.661438 13452 webserver.cc:460] Webserver started at http://127.0.97.66:40431/ using document root <none> and password file <none>
I20260620 15:19:16.662110 13452 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:16.662261 13452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:19:16 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:19:16.682961 13452 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:19:16.686543 13477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:16.688194 13452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:19:16.688354 13452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "af7efe03f44049b4ace3ba3d67b22493"
format_stamp: "Formatted at 2026-06-20 15:19:13 on dist-test-slave-7s64"
server_key: "b194638a63723febe99769e5a99daed0"
server_key_iv: "d62aa0673365c2d5d47596df49c7e5cc"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:16.689851 13452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:16.756138 13452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:16.761660 13452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:16.762012 13452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:16.763301 13452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:16.765115 13452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:16.765264 13452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 15:19:16.765424 13452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:16.765565 13452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:16.815301 13452 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:34847
I20260620 15:19:16.815371 13591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:34847 every 8 connection(s)
Jun 20 15:19:16 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781968756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:16.817296 13452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:19:16.821972 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 13452
I20260620 15:19:16.822711 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:46103/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:40503
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:37675
--webserver_port=37903
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:46103/kms with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:19:16 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:16.848156 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:16.821099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37675 (local address 127.0.97.66:53669)
0620 15:19:16.825970 (+ 4871us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:16.826029 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:16.828368 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:16.829831 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:16.829844 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:16.830465 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:16.831261 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:16.831286 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:16.832857 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:16.832863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:16.834868 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:16.834883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:16.836169 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:16.836176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:16.836439 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:16.837201 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:16.837230 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:16.839832 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:16.843655 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:16.843663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:16.843678 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:16.844240 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:16.844704 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:16.844710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:16.844713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:16.844929 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:16.845659 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:16.845666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:16.846042 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 15:19:16.846327 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 15:19:16.846834 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4567,"thread_start_us":5817,"threads_started":1}
I20260620 15:19:16.849308 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:16.820231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:53669 (local address 127.0.97.126:37675)
0620 15:19:16.829046 (+ 8815us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:16.829053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:16.829072 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:19:16.829133 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:16.829137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:16.829225 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:16.829340 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:16.831467 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:16.832674 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:16.835042 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:16.835716 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:16.835812 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:16.840075 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:16.840081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:16.840086 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:16.840122 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:16.843476 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:16.844383 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:16.844388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:16.844392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:16.844569 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:16.845066 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:16.845071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:16.845074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:16.845427 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:16.845576 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:16.848756 (+ 3180us) server_negotiation.cc:299] Negotiation successful
0620 15:19:16.848975 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8686,"thread_start_us":9868,"threads_started":1}
I20260620 15:19:16.851370 13592 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37675
I20260620 15:19:16.851742 13592 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:16.852712 13592 heartbeater.cc:511] Master 127.0.97.126:37675 requested a full tablet report, sending...
I20260620 15:19:16.854735 13251 ts_manager.cc:194] Registered new tserver with Master: af7efe03f44049b4ace3ba3d67b22493 (127.0.97.66:34847)
I20260620 15:19:16.856936 13251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:53669
I20260620 15:19:17.221590 13448 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:17.221992 13448 heartbeater.cc:511] Master 127.0.97.126:37675 requested a full tablet report, sending...
I20260620 15:19:17.223150 13251 ts_manager.cc:194] Registered new tserver with Master: 7b2c0307fa5942c2b5b0d02368a60563 (127.0.97.65:36451)
I20260620 15:19:17.224308 13251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:37021
Jun 20 15:19:17 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
W20260620 15:19:17.281641 13596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:17.282001 13596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:17.282145 13596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:17.292869 13596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:17.293011 13596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:17.293210 13596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:19:17.304446 13596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:46103/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:40503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=37903
--tserver_master_addrs=127.0.97.126:37675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:17.305816 13596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:17.307083 13596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:17.322975 13602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:17.323496 13603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:17.326629 13605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:17.328013 13596 server_base.cc:1034] running on GCE node
I20260620 15:19:17.328718 13596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:17.330873 13596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:17.332109 13596 hybrid_clock.cc:648] HybridClock initialized: now 1781968757332057 us; error 57 us; skew 500 ppm
I20260620 15:19:17.332425 13596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:17 dist-test-slave-7s64 krb5kdc[12241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:17.335567 13596 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:19:17.340904 13596 webserver.cc:460] Webserver started at http://127.0.97.67:37903/ using document root <none> and password file <none>
I20260620 15:19:17.341738 13596 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:17.341950 13596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:19:17 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:19:17.364308 13596 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:19:17.368214 13613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:17.369985 13596 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:19:17.370221 13596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "cbb6cb83e03740f19bbf7686a18ccafb"
format_stamp: "Formatted at 2026-06-20 15:19:14 on dist-test-slave-7s64"
server_key: "0db24d2cea8b1c3ff7d5302472433ee9"
server_key_iv: "2de0fa0d725c59f8c27d5ca11c4e9bbf"
server_key_version: "kuduclusterkey@0"
I20260620 15:19:17.371735 13596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:17.392052 13596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:17.397502 13596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:17.397858 13596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:17.399196 13596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:17.401126 13596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:17.401278 13596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 15:19:17.401434 13596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:17.401580 13596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:17.445515 13596 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:40503
I20260620 15:19:17.445572 13725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:40503 every 8 connection(s)
I20260620 15:19:17.447527 13596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:19:17.453656 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 13596
Jun 20 15:19:17 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:17.499805 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.454622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.0.97.126:37675)
0620 15:19:17.478552 (+ 23930us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:17.478558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:17.478577 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:19:17.478642 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:17.478648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:17.478744 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:17.481055 (+ 2311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:17.481213 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.482270 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.487566 (+ 5296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.488176 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.488296 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.491601 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:17.491607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:17.491612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:17.491651 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:17.495446 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:17.496419 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:17.496425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:17.496428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:17.496860 (+ 432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:17.497316 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:17.497323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:17.497327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:17.497663 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:17.499013 (+ 1350us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:17.499188 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 15:19:17.499405 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23826,"thread_start_us":23341,"threads_started":1}
I20260620 15:19:17.507692 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.451588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:41117 (local address 127.0.97.126:37675)
0620 15:19:17.478222 (+ 26634us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:17.478229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:17.483515 (+ 5286us) server_negotiation.cc:407] Connection header received
0620 15:19:17.487818 (+ 4303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:17.487824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:17.487925 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:17.488041 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:17.490921 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.492128 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.494662 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.495400 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.495510 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.499947 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:17.499953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:17.499958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:17.500002 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:17.504117 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:17.505051 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:17.505056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:17.505059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:17.505190 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:17.505690 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:17.505695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:17.505699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:17.506020 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:17.506163 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:17.507103 (+ 940us) server_negotiation.cc:299] Negotiation successful
0620 15:19:17.507307 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26494,"thread_start_us":29639,"threads_started":1}
I20260620 15:19:17.509886 13728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.450950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37675 (local address 127.0.97.67:41117)
0620 15:19:17.483007 (+ 32057us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:17.483074 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:17.487375 (+ 4301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:17.489193 (+ 1818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:17.489207 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:17.489795 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:17.490681 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.490707 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.492309 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.492314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.494490 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.494504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.495949 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.495958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.496245 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.496979 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:17.497008 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:17.499629 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:17.504272 (+ 4643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:17.504281 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:17.504299 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:17.504911 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:17.505334 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:17.505339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:17.505342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:17.505564 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:17.506257 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:17.506265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:17.506687 (+ 422us) client_negotiation.cc:769] Sending connection context
0620 15:19:17.507938 (+ 1251us) client_negotiation.cc:240] Negotiation successful
0620 15:19:17.508480 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31717,"thread_start_us":31495,"threads_started":1}
I20260620 15:19:17.511518 13726 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37675
I20260620 15:19:17.511853 13726 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:17.512784 13726 heartbeater.cc:511] Master 127.0.97.126:37675 requested a full tablet report, sending...
I20260620 15:19:17.515991 13251 ts_manager.cc:194] Registered new tserver with Master: cbb6cb83e03740f19bbf7686a18ccafb (127.0.97.67:40503)
I20260620 15:19:17.517091 13251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:41117
I20260620 15:19:17.528631 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:19:17 dist-test-slave-7s64 krb5kdc[12241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260620 15:19:17.562841 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.544790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.0.97.126:37675)
0620 15:19:17.545024 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:17.545031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:17.545516 (+ 485us) server_negotiation.cc:407] Connection header received
0620 15:19:17.545814 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:17.545821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:17.545923 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:17.546021 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:17.548637 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.549758 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.551812 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.552415 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.552537 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.556435 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:17.556443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:17.556459 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:17.556528 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:17.559781 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:17.560462 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:17.560468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:17.560471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:17.560634 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:17.561095 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:17.561100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:17.561104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:17.561464 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:17.561686 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:17.562265 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 15:19:17.562509 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 15:19:17.570144 13251 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:19:17.582854 13251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:19:17.721028 13743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.636468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:34847 (local address 127.0.0.1:59816)
0620 15:19:17.693797 (+ 57329us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:17.693863 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:17.694086 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:17.698401 (+ 4315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:17.698409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:17.698438 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:17.698974 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.698991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.703841 (+ 4850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.703850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.707406 (+ 3556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.707427 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.720027 (+ 12600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.720035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.720255 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.720309 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 15:19:17.720444 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 15:19:17.720659 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56938,"thread_start_us":60031,"threads_started":1}
I20260620 15:19:17.721948 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.637031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.0.97.66:34847)
0620 15:19:17.694808 (+ 57777us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:17.694822 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:17.694858 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:19:17.694976 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:17.694987 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:17.695314 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:17.695525 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:17.701596 (+ 6071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.703627 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.710975 (+ 7348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.713064 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.713220 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.713499 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:17.721458 (+ 7959us) server_negotiation.cc:299] Negotiation successful
0620 15:19:17.721652 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57555,"thread_start_us":60357,"threads_started":1}
I20260620 15:19:17.723729 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.646821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36451 (local address 127.0.0.1:52266)
0620 15:19:17.713974 (+ 67153us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:17.713994 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:17.714195 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:17.715074 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:17.715077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:17.715095 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:17.715374 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.715381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.717831 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.717836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.719814 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.719828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.722812 (+ 2984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.722816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.723077 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.723108 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:19:17.723204 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:19:17.723317 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67027,"mutex_wait_us":41,"thread_start_us":50422,"threads_started":1}
I20260620 15:19:17.724404 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.647225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.0.97.65:36451)
0620 15:19:17.698122 (+ 50897us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:17.698133 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:17.714092 (+ 15959us) server_negotiation.cc:407] Connection header received
0620 15:19:17.714384 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:17.714394 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:17.714651 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:17.714809 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:17.715539 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.717658 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.719972 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.721404 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.721515 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.721731 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:17.723926 (+ 2195us) server_negotiation.cc:299] Negotiation successful
0620 15:19:17.724079 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50688,"thread_start_us":50909,"threads_started":1}
I20260620 15:19:17.727836 13527 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea1504757ba4c7293f355d6a2f90809 (DEFAULT_TABLE table=test-table [id=0b821c4da340454eba9439c84941e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:17.731052 13383 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea1504757ba4c7293f355d6a2f90809 (DEFAULT_TABLE table=test-table [id=0b821c4da340454eba9439c84941e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:17.732409 13383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea1504757ba4c7293f355d6a2f90809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:17.737098 13527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea1504757ba4c7293f355d6a2f90809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:17.738580 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.646424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40503 (local address 127.0.0.1:44340)
0620 15:19:17.696195 (+ 49771us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:17.696218 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:17.696342 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:17.700886 (+ 4544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:17.700893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:17.700927 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:17.701345 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.701358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.716023 (+ 14665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.716029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.718110 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.718125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.722806 (+ 4681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.722812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.723363 (+ 551us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.723397 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:19:17.723469 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 15:19:17.723590 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49627,"mutex_wait_us":322,"thread_start_us":67270,"threads_started":1}
I20260620 15:19:17.743350 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.647195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.0.97.67:40503)
0620 15:19:17.699818 (+ 52623us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:17.699830 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:17.699870 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:19:17.699984 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:17.699994 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:17.700287 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:17.700523 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:17.707684 (+ 7161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.709833 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.718320 (+ 8487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.719799 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.723098 (+ 3299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.741638 (+ 18540us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:17.741723 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 15:19:17.741906 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52431,"thread_start_us":52258,"threads_started":1}
I20260620 15:19:17.747080 13661 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea1504757ba4c7293f355d6a2f90809 (DEFAULT_TABLE table=test-table [id=0b821c4da340454eba9439c84941e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:17.748452 13661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea1504757ba4c7293f355d6a2f90809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:17.777848 13750 tablet_bootstrap.cc:492] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: Bootstrap starting.
I20260620 15:19:17.778708 13751 tablet_bootstrap.cc:492] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: Bootstrap starting.
I20260620 15:19:17.778777 13749 tablet_bootstrap.cc:492] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: Bootstrap starting.
I20260620 15:19:17.781500 13750 tablet_bootstrap.cc:654] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:17.782756 13750 log.cc:824] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:17.783689 13749 tablet_bootstrap.cc:654] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:17.784878 13749 log.cc:824] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:17.786237 13751 tablet_bootstrap.cc:654] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:17.789985 13751 log.cc:824] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:17.791992 13750 tablet_bootstrap.cc:492] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: No bootstrap required, opened a new log
I20260620 15:19:17.792258 13750 ts_tablet_manager.cc:1397] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20260620 15:19:17.798411 13749 tablet_bootstrap.cc:492] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: No bootstrap required, opened a new log
I20260620 15:19:17.798684 13749 ts_tablet_manager.cc:1397] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s
I20260620 15:19:17.806602 13750 raft_consensus.cc:348] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7efe03f44049b4ace3ba3d67b22493" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 34847 } } peers { permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 } } peers { permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 } }
I20260620 15:19:17.807039 13750 raft_consensus.cc:374] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:17.807138 13750 raft_consensus.cc:729] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2c0307fa5942c2b5b0d02368a60563, State: Initialized, Role: FOLLOWER
I20260620 15:19:17.807852 13750 consensus_queue.cc:260] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7efe03f44049b4ace3ba3d67b22493" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 34847 } } peers { permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 } } peers { permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 } }
I20260620 15:19:17.809870 13751 tablet_bootstrap.cc:492] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: No bootstrap required, opened a new log
I20260620 15:19:17.810149 13751 ts_tablet_manager.cc:1397] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.004s
I20260620 15:19:17.813328 13749 raft_consensus.cc:348] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7efe03f44049b4ace3ba3d67b22493" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 34847 } } peers { permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 } } peers { permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 } }
I20260620 15:19:17.813791 13749 raft_consensus.cc:374] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:17.813925 13749 raft_consensus.cc:729] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af7efe03f44049b4ace3ba3d67b22493, State: Initialized, Role: FOLLOWER
I20260620 15:19:17.814613 13749 consensus_queue.cc:260] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7efe03f44049b4ace3ba3d67b22493" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 34847 } } peers { permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 } } peers { permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 } }
I20260620 15:19:17.815949 13750 ts_tablet_manager.cc:1428] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: Time spent starting tablet: real 0.024s user 0.011s sys 0.009s
I20260620 15:19:17.816293 13448 heartbeater.cc:503] Master 127.0.97.126:37675 was elected leader, sending a full tablet report...
I20260620 15:19:17.843631 13751 raft_consensus.cc:348] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7efe03f44049b4ace3ba3d67b22493" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 34847 } } peers { permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 } } peers { permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 } }
I20260620 15:19:17.844187 13751 raft_consensus.cc:374] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:17.844290 13751 raft_consensus.cc:729] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb6cb83e03740f19bbf7686a18ccafb, State: Initialized, Role: FOLLOWER
I20260620 15:19:17.849116 13756 raft_consensus.cc:482] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:19:17.849220 13592 heartbeater.cc:503] Master 127.0.97.126:37675 was elected leader, sending a full tablet report...
I20260620 15:19:17.849417 13756 raft_consensus.cc:504] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7efe03f44049b4ace3ba3d67b22493" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 34847 } } peers { permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 } } peers { permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 } }
I20260620 15:19:17.850533 13749 ts_tablet_manager.cc:1428] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: Time spent starting tablet: real 0.052s user 0.038s sys 0.011s
I20260620 15:19:17.855297 13751 consensus_queue.cc:260] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7efe03f44049b4ace3ba3d67b22493" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 34847 } } peers { permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 } } peers { permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 } }
I20260620 15:19:17.857333 13756 leader_election.cc:288] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbb6cb83e03740f19bbf7686a18ccafb (127.0.97.67:40503), 7b2c0307fa5942c2b5b0d02368a60563 (127.0.97.65:36451)
I20260620 15:19:17.875165 13726 heartbeater.cc:503] Master 127.0.97.126:37675 was elected leader, sending a full tablet report...
I20260620 15:19:17.876154 13751 ts_tablet_manager.cc:1428] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: Time spent starting tablet: real 0.066s user 0.011s sys 0.023s
I20260620 15:19:17.884352 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.859385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:36903 (local address 127.0.97.65:36451)
0620 15:19:17.859787 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:17.859794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:17.874406 (+ 14612us) server_negotiation.cc:407] Connection header received
0620 15:19:17.874584 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:17.874590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:17.874680 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:17.874779 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:17.878462 (+ 3683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.879533 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.882054 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.883180 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.883274 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.883372 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:17.883950 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 15:19:17.884081 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260620 15:19:17.885191 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.858345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36451 (local address 127.0.97.66:36903)
0620 15:19:17.874313 (+ 15968us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:17.874332 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:17.874454 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:17.876951 (+ 2497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:17.876956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:17.876991 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:17.878245 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.878256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.879737 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.879743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.881882 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.881897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.883583 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.883589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.883762 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.883797 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:19:17.884732 (+ 935us) client_negotiation.cc:240] Negotiation successful
0620 15:19:17.884865 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15814,"thread_start_us":14890,"threads_started":1}
I20260620 15:19:17.886261 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea1504757ba4c7293f355d6a2f90809" candidate_uuid: "af7efe03f44049b4ace3ba3d67b22493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b2c0307fa5942c2b5b0d02368a60563" is_pre_election: true
I20260620 15:19:17.886863 13403 raft_consensus.cc:2454] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7efe03f44049b4ace3ba3d67b22493 in term 0.
I20260620 15:19:17.887421 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.859030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41685 (local address 127.0.97.67:40503)
0620 15:19:17.859615 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:17.859622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:17.873700 (+ 14078us) server_negotiation.cc:407] Connection header received
0620 15:19:17.873933 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:17.873940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:17.874038 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:17.874145 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:17.878706 (+ 4561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.879847 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.884955 (+ 5108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.886227 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.886332 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.886435 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:17.887052 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 15:19:17.887187 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429}
I20260620 15:19:17.887605 13478 leader_election.cc:302] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b2c0307fa5942c2b5b0d02368a60563, af7efe03f44049b4ace3ba3d67b22493; no voters:
I20260620 15:19:17.888063 13757 raft_consensus.cc:2781] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:19:17.888156 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.857717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40503 (local address 127.0.97.66:41685)
0620 15:19:17.873575 (+ 15858us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:17.873607 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:17.873781 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:17.874328 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:17.874333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:17.874362 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:17.877946 (+ 3584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.877961 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.882454 (+ 4493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.882461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.884765 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:17.884781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.886644 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.886651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:17.886841 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.886877 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:19:17.887641 (+ 764us) client_negotiation.cc:240] Negotiation successful
0620 15:19:17.887785 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15678,"thread_start_us":19023,"threads_started":1}
I20260620 15:19:17.888247 13757 raft_consensus.cc:482] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:19:17.888407 13757 raft_consensus.cc:3037] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:17.889099 13681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea1504757ba4c7293f355d6a2f90809" candidate_uuid: "af7efe03f44049b4ace3ba3d67b22493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" is_pre_election: true
I20260620 15:19:17.889566 13681 raft_consensus.cc:2454] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af7efe03f44049b4ace3ba3d67b22493 in term 0.
I20260620 15:19:17.891289 13757 raft_consensus.cc:504] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7efe03f44049b4ace3ba3d67b22493" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 34847 } } peers { permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 } } peers { permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 } }
I20260620 15:19:17.891974 13757 leader_election.cc:288] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [CANDIDATE]: Term 1 election: Requested vote from peers cbb6cb83e03740f19bbf7686a18ccafb (127.0.97.67:40503), 7b2c0307fa5942c2b5b0d02368a60563 (127.0.97.65:36451)
I20260620 15:19:17.892284 13681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea1504757ba4c7293f355d6a2f90809" candidate_uuid: "af7efe03f44049b4ace3ba3d67b22493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb6cb83e03740f19bbf7686a18ccafb"
I20260620 15:19:17.892472 13681 raft_consensus.cc:3037] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:17.892438 13403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea1504757ba4c7293f355d6a2f90809" candidate_uuid: "af7efe03f44049b4ace3ba3d67b22493" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b2c0307fa5942c2b5b0d02368a60563"
I20260620 15:19:17.892674 13403 raft_consensus.cc:3037] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:17.895561 13681 raft_consensus.cc:2454] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7efe03f44049b4ace3ba3d67b22493 in term 1.
I20260620 15:19:17.895998 13479 leader_election.cc:302] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af7efe03f44049b4ace3ba3d67b22493, cbb6cb83e03740f19bbf7686a18ccafb; no voters:
I20260620 15:19:17.896013 13403 raft_consensus.cc:2454] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af7efe03f44049b4ace3ba3d67b22493 in term 1.
I20260620 15:19:17.896267 13757 raft_consensus.cc:2781] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:19:17.896538 13757 raft_consensus.cc:686] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 1 LEADER]: Becoming Leader. State: Replica: af7efe03f44049b4ace3ba3d67b22493, State: Running, Role: LEADER
I20260620 15:19:17.896956 13757 consensus_queue.cc:237] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7efe03f44049b4ace3ba3d67b22493" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 34847 } } peers { permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 } } peers { permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 } }
I20260620 15:19:17.901820 13251 catalog_manager.cc:5495] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 reported cstate change: term changed from 0 to 1, leader changed from <none> to af7efe03f44049b4ace3ba3d67b22493 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "af7efe03f44049b4ace3ba3d67b22493" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af7efe03f44049b4ace3ba3d67b22493" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 34847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 } health_report { overall_health: UNKNOWN } } }
I20260620 15:19:17.921844 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:17.915762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.0.97.66:34847)
0620 15:19:17.915947 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:17.915953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:17.915972 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:19:17.916128 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:17.916134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:17.916263 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:17.916412 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:19:17.916917 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.918048 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.919328 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:17.919969 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:17.920099 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:17.920594 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:19:17.921244 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:19:17.921382 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:17.921477 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 15:19:17.921575 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:19:17.930809 13403 raft_consensus.cc:1261] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563 [term 1 FOLLOWER]: Refusing update from remote peer af7efe03f44049b4ace3ba3d67b22493: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:19:17.931061 13681 raft_consensus.cc:1261] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb [term 1 FOLLOWER]: Refusing update from remote peer af7efe03f44049b4ace3ba3d67b22493: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:19:17.931819 13756 consensus_queue.cc:1035] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b2c0307fa5942c2b5b0d02368a60563" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:19:17.931898 13449 tablet.cc:2369] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:19:17.932235 13757 consensus_queue.cc:1035] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb6cb83e03740f19bbf7686a18ccafb" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:19:17.983549 13250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:19:17.983994 13250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:19:17.990334 13250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b39801a8a25e4d8392d586dc8484bcad: Sending DeleteTablet for 3 replicas of tablet 8ea1504757ba4c7293f355d6a2f90809
I20260620 15:19:17.992030 13527 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea1504757ba4c7293f355d6a2f90809 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:59816
I20260620 15:19:17.992858 13661 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea1504757ba4c7293f355d6a2f90809 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44340
I20260620 15:19:17.993718 13383 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea1504757ba4c7293f355d6a2f90809 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:52266
I20260620 15:19:17.998749 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 13288
I20260620 15:19:18.011067 13779 tablet_replica.cc:330] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: stopping tablet replica
I20260620 15:19:18.011526 13779 raft_consensus.cc:2229] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:19:18.012029 13779 raft_consensus.cc:2258] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:18.019554 13781 tablet_replica.cc:330] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: stopping tablet replica
I20260620 15:19:18.020076 13781 raft_consensus.cc:2229] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:19:18.021087 13781 raft_consensus.cc:2258] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:18.021590 13780 tablet_replica.cc:330] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: stopping tablet replica
I20260620 15:19:18.022111 13779 ts_tablet_manager.cc:1905] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:18.023728 13781 ts_tablet_manager.cc:1905] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:18.024150 13780 raft_consensus.cc:2229] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:19:18.024659 13780 raft_consensus.cc:2258] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:18.026468 13780 ts_tablet_manager.cc:1905] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:18.029395 13779 ts_tablet_manager.cc:1918] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:18.029563 13779 log.cc:1196] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/8ea1504757ba4c7293f355d6a2f90809
I20260620 15:19:18.030102 13779 ts_tablet_manager.cc:1939] T 8ea1504757ba4c7293f355d6a2f90809 P af7efe03f44049b4ace3ba3d67b22493: Deleting consensus metadata
I20260620 15:19:18.038760 13781 ts_tablet_manager.cc:1918] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:18.040596 13780 ts_tablet_manager.cc:1918] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:18.040339 13748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.006116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.0.97.65:36451)
0620 15:19:18.006291 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:18.006297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:18.022619 (+ 16322us) server_negotiation.cc:407] Connection header received
0620 15:19:18.023046 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:18.023051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:18.023128 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:18.023203 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:18.025061 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:18.026149 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:18.027968 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:18.028726 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:18.028829 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:18.032781 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:18.032794 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:18.032810 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:18.032857 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:18.036845 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:18.037698 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:18.037707 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:18.037711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:18.037870 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:18.038306 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:18.038312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:18.038316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:18.038780 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:18.039062 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:18.039656 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 15:19:18.039880 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:19:18.041160 13781 log.cc:1196] T 8ea1504757ba4c7293f355d6a2f90809 P 7b2c0307fa5942c2b5b0d02368a60563: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/8ea1504757ba4c7293f355d6a2f90809
I20260620 15:19:18.041184 13237 catalog_manager.cc:4841] TS af7efe03f44049b4ace3ba3d67b22493 (127.0.97.66:34847): tablet 8ea1504757ba4c7293f355d6a2f90809 (table test-table [id=0b821c4da340454eba9439c84941e214]) successfully deleted
I20260620 15:19:18.041649 13443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:18.046525 13780 log.cc:1196] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/8ea1504757ba4c7293f355d6a2f90809
I20260620 15:19:18.047129 13780 ts_tablet_manager.cc:1939] T 8ea1504757ba4c7293f355d6a2f90809 P cbb6cb83e03740f19bbf7686a18ccafb: Deleting consensus metadata
I20260620 15:19:18.048846 13236 catalog_manager.cc:4841] TS cbb6cb83e03740f19bbf7686a18ccafb (127.0.97.67:40503): tablet 8ea1504757ba4c7293f355d6a2f90809 (table test-table [id=0b821c4da340454eba9439c84941e214]) successfully deleted
I20260620 15:19:18.193397 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 13288
I20260620 15:19:18.216055 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 13452
W20260620 15:19:18.216516 13235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:36451 (error 108)
I20260620 15:19:18.229874 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.229453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36451 (local address 127.0.0.1:52276)
0620 15:19:18.229607 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:18.229715 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 15:19:18.230343 13235 catalog_manager.cc:4584] TS 7b2c0307fa5942c2b5b0d02368a60563 (127.0.97.65:36451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ea1504757ba4c7293f355d6a2f90809: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36451: connect: Connection refused (error 111)
I20260620 15:19:18.234671 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.217615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.0.97.66:34847)
0620 15:19:18.217800 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:18.217807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:18.217824 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:19:18.217885 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:18.217890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:18.217972 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:18.218079 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:18.219893 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:18.221397 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:18.223308 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:18.223937 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:18.224066 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:18.227332 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:18.227350 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:18.227370 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:18.227425 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:18.231516 (+ 4091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:18.232336 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:18.232345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:18.232349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:18.232542 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:18.232994 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:18.233000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:18.233003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:18.233452 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:18.233784 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:18.234075 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:19:18.234300 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:19:18.235643 13587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:18.284070 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.283544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36451 (local address 127.0.0.1:52282)
0620 15:19:18.283774 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:18.283885 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:19:18.285197 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.284784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36451 (local address 127.0.0.1:52290)
0620 15:19:18.284942 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:18.285026 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 15:19:18.345845 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.345312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36451 (local address 127.0.0.1:52306)
0620 15:19:18.345531 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:18.345638 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 15:19:18.346926 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.346549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36451 (local address 127.0.0.1:52320)
0620 15:19:18.346704 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:18.346787 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:19:18.350903 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 13452
I20260620 15:19:18.369761 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 13596
I20260620 15:19:18.382344 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.370432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.0.97.67:40503)
0620 15:19:18.370577 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:18.370581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:18.370600 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:19:18.370756 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:18.370760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:18.370831 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:18.370911 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:18.372105 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:18.372898 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:18.374079 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:18.374486 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:18.374575 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:18.377163 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:18.377174 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:18.377187 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:18.377216 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:18.379971 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:18.380620 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:18.380627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:18.380629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:18.380755 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:18.381048 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:18.381051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:18.381053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:18.381404 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:18.381610 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:18.381841 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 15:19:18.381981 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 15:19:18.383006 13721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:18.427526 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.427010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36451 (local address 127.0.0.1:52322)
0620 15:19:18.427222 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:18.427334 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 15:19:18.428623 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.428173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36451 (local address 127.0.0.1:52332)
0620 15:19:18.428375 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:18.428458 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 15:19:18.474004 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 13596
I20260620 15:19:18.490679 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 13218
I20260620 15:19:18.491933 13281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:18.603999 13746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:18.603460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36451 (local address 127.0.0.1:52346)
0620 15:19:18.603665 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:18.603748 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 15:19:18.604249 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 13218
I20260620 15:19:18.625841 389 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 15:19:18.960098 389 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 15:19:18.960331 389 mini_ranger.cc:66] Stopping Ranger...
I20260620 15:19:19.304495 389 mini_ranger.cc:68] Stopped Ranger
2026-06-20 15:19:19.304 UTC [12271] LOG: received smart shutdown request
2026-06-20 15:19:19.307 UTC [12271] LOG: background worker "logical replication launcher" (PID 12280) exited with exit code 1
2026-06-20 15:19:19.308 UTC [12275] LOG: shutting down
2026-06-20 15:19:19.445 UTC [12271] LOG: database system is shut down
2026-06-20T15:19:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50364 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:19:19 dist-test-slave-7s64 krb5kdc[13796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:19:19 dist-test-slave-7s64 krb5kdc[13796](info): set up 2 sockets
Jun 20 15:19:19 dist-test-slave-7s64 krb5kdc[13796](info): commencing operation
krb5kdc: starting...
W20260620 15:19:20.652849 389 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:19:20 dist-test-slave-7s64 krb5kdc[13796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:19:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:19:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:19:20.865792 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39641
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:39641,127.0.97.125:34011,127.0.97.124:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:21.187534 13809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:21.187795 13809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:21.187881 13809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:21.194880 13809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:19:21.194975 13809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:21.195020 13809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:21.195068 13809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:19:21.195104 13809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:19:21.204438 13809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:39641,127.0.97.125:34011,127.0.97.124:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:21.205649 13809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:21.206843 13809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:21.218534 13814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:21.218566 13817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:21.218593 13815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:21.219609 13809 server_base.cc:1034] running on GCE node
I20260620 15:19:21.220329 13809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:21.221675 13809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:21.222872 13809 hybrid_clock.cc:648] HybridClock initialized: now 1781968761222831 us; error 36 us; skew 500 ppm
I20260620 15:19:21.223212 13809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:21 dist-test-slave-7s64 krb5kdc[13796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:21.226271 13809 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:19:21.227705 13809 webserver.cc:460] Webserver started at http://127.0.97.126:36547/ using document root <none> and password file <none>
I20260620 15:19:21.228353 13809 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:21.228562 13809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:21.228860 13809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:21.231688 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "1272d5a162944602a76f541a101cbb7f"
format_stamp: "Formatted at 2026-06-20 15:19:21 on dist-test-slave-7s64"
server_key: "6811186e3e61612b1278ece55ab4eae5"
server_key_iv: "2a635760db78b5ab078624e81e2dcbff"
server_key_version: "encryptionkey@0"
I20260620 15:19:21.232473 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "1272d5a162944602a76f541a101cbb7f"
format_stamp: "Formatted at 2026-06-20 15:19:21 on dist-test-slave-7s64"
server_key: "6811186e3e61612b1278ece55ab4eae5"
server_key_iv: "2a635760db78b5ab078624e81e2dcbff"
server_key_version: "encryptionkey@0"
I20260620 15:19:21.237283 13809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260620 15:19:21.240615 13824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:21.241866 13809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:19:21.242043 13809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "1272d5a162944602a76f541a101cbb7f"
format_stamp: "Formatted at 2026-06-20 15:19:21 on dist-test-slave-7s64"
server_key: "6811186e3e61612b1278ece55ab4eae5"
server_key_iv: "2a635760db78b5ab078624e81e2dcbff"
server_key_version: "encryptionkey@0"
I20260620 15:19:21.242233 13809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:21.267897 13809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:21.271626 13809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:21.271909 13809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:21.291653 13809 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:39641
I20260620 15:19:21.291690 13875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:39641 every 8 connection(s)
I20260620 15:19:21.293013 13809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:19:21.294669 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 13809
I20260620 15:19:21.294888 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:19:21.295696 389 external_mini_cluster.cc:1428] Setting key 423b3244144b4b013852c6cf709ec0cf
I20260620 15:19:21.297333 13876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:21.306026 13876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.125@KRBTEST.COM" created.
Jun 20 15:19:21 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:21.329829 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.309289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.0.97.126:39641)
0620 15:19:21.310190 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:21.310206 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:21.310742 (+ 536us) server_negotiation.cc:407] Connection header received
0620 15:19:21.312680 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:21.312692 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:21.313005 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:21.313238 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:21.314314 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.316326 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.317774 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.318387 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.318522 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:21.322013 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:21.322025 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:21.322043 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:21.322083 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:21.325955 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:21.326764 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:21.326770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:21.326772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:21.326902 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:21.327365 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:21.327368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:21.327370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:21.327795 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:21.327937 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:21.328621 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 15:19:21.328839 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":413,"threads_started":1}
I20260620 15:19:21.329856 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.309289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39641 (local address 127.0.0.1:44340)
0620 15:19:21.310559 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.310601 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:21.312273 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:21.313264 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:21.313269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:21.313798 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:21.314105 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:21.314120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.316480 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.316527 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:21.317646 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:21.317657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.318577 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.318582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:21.318694 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:21.319328 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:21.319349 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:21.321779 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:21.326080 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:21.326089 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:21.326098 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:21.326626 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:21.327010 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:21.327017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:21.327022 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:21.327250 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:21.327941 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:21.327951 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:21.328198 (+ 247us) client_negotiation.cc:769] Sending connection context
0620 15:19:21.328451 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 15:19:21.329052 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":84,"spinlock_wait_cycles":57088,"thread_start_us":799,"threads_started":1}
I20260620 15:19:21.332628 13876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:19:21.333729 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.333252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34232)
0620 15:19:21.333446 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.333580 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 15:19:21.334007 13825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
I20260620 15:19:21.335623 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.335172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34244)
0620 15:19:21.335356 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.335448 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:19:21.335899 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:34011: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab.
I20260620 15:19:21.345474 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:34011
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:39641,127.0.97.125:34011,127.0.97.124:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:19:21.385192 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } attempt: 1
I20260620 15:19:21.386457 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.385855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34250)
0620 15:19:21.386138 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.386287 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 15:19:21.387567 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.387108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34262)
0620 15:19:21.387317 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.387407 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 15:19:21.387784 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:34011: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
I20260620 15:19:21.456056 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } attempt: 2
I20260620 15:19:21.457315 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.456786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34278)
0620 15:19:21.457018 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.457144 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 15:19:21.458350 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.457940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34282)
0620 15:19:21.458108 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.458191 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:19:21.458546 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:34011: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
I20260620 15:19:21.549808 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } attempt: 3
I20260620 15:19:21.551110 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.550563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34288)
0620 15:19:21.550831 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.550947 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 15:19:21.552109 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.551725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34292)
0620 15:19:21.551869 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.551961 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:19:21.552307 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:34011: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
W20260620 15:19:21.660009 13883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:21.660283 13883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:21.660369 13883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:21.667147 13883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:19:21.667232 13883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:21.667275 13883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:21.667331 13883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:19:21.667379 13883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:19:21.676574 13883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:39641,127.0.97.125:34011,127.0.97.124:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:34011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:21.677687 13883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:21.678820 13883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:21.689837 13891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:21.689890 13889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:21.689916 13888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:21.690840 13883 server_base.cc:1034] running on GCE node
I20260620 15:19:21.691634 13883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:21.693120 13883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:21.694355 13883 hybrid_clock.cc:648] HybridClock initialized: now 1781968761694294 us; error 48 us; skew 500 ppm
I20260620 15:19:21.694671 13883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:19:21.695551 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } attempt: 4
I20260620 15:19:21.696820 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.696151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34304)
0620 15:19:21.696521 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.696643 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
Jun 20 15:19:21 dist-test-slave-7s64 krb5kdc[13796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:21.697993 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.697588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34320)
0620 15:19:21.697731 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.697814 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:19:21.698241 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:34011: Network error: Client connection negotiation failed: client connection to 127.0.97.125:34011: connect: Connection refused (error 111)
I20260620 15:19:21.698356 13883 init.cc:377] Logged in from keytab as kudu/127.0.97.125@KRBTEST.COM (short username kudu)
I20260620 15:19:21.699723 13883 webserver.cc:460] Webserver started at http://127.0.97.125:44427/ using document root <none> and password file <none>
I20260620 15:19:21.700374 13883 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:21.700549 13883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:21.700845 13883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:21.703687 13883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/data/instance:
uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7"
format_stamp: "Formatted at 2026-06-20 15:19:21 on dist-test-slave-7s64"
server_key: "549f37ac3bbcb73fe34a27bd755d9270"
server_key_iv: "ad7e663d5b4b421cdcd2369ded092b3f"
server_key_version: "encryptionkey@0"
I20260620 15:19:21.704466 13883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/wal/instance:
uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7"
format_stamp: "Formatted at 2026-06-20 15:19:21 on dist-test-slave-7s64"
server_key: "549f37ac3bbcb73fe34a27bd755d9270"
server_key_iv: "ad7e663d5b4b421cdcd2369ded092b3f"
server_key_version: "encryptionkey@0"
I20260620 15:19:21.709363 13883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:19:21.712471 13898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:21.713866 13883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:19:21.714046 13883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/wal
uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7"
format_stamp: "Formatted at 2026-06-20 15:19:21 on dist-test-slave-7s64"
server_key: "549f37ac3bbcb73fe34a27bd755d9270"
server_key_iv: "ad7e663d5b4b421cdcd2369ded092b3f"
server_key_version: "encryptionkey@0"
I20260620 15:19:21.714244 13883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:21.740252 13883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:21.744611 13883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:21.744882 13883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:21.762782 13883 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:34011
I20260620 15:19:21.762811 13949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:34011 every 8 connection(s)
I20260620 15:19:21.764021 13883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
I20260620 15:19:21.767505 13950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:21.773554 13950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:19:21.774294 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 13883
I20260620 15:19:21.774478 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-1/wal/instance
I20260620 15:19:21.774889 389 external_mini_cluster.cc:1428] Setting key 7eb51d8611969d15c9600d975f77b85a
Jun 20 15:19:21 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:21.796173 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.776077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.0.97.126:39641)
0620 15:19:21.776320 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:21.776329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:21.777489 (+ 1160us) server_negotiation.cc:407] Connection header received
0620 15:19:21.780248 (+ 2759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:21.780252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:21.780317 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:21.780402 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:21.782522 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.783218 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.784898 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.785661 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.785806 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:21.789087 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:21.789096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:21.789113 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:21.789157 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:21.792813 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:21.793856 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:21.793859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:21.793862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:21.793973 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:21.794492 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:21.794495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:21.794497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:21.794750 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:21.794856 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:21.795796 (+ 940us) server_negotiation.cc:299] Negotiation successful
0620 15:19:21.795950 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
WARNING: no policy specified for kudu/127.0.97.124@KRBTEST.COM; defaulting to no policy
I20260620 15:19:21.797742 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.776529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39641 (local address 127.0.0.1:44356)
0620 15:19:21.777317 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.777364 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:21.779957 (+ 2593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:21.780690 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:21.780704 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:21.781442 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:21.782212 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:21.782239 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.783397 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.783404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:21.784761 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:21.784773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.785865 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.785870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:21.786036 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:21.786689 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:21.786710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:21.788860 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:21.793001 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:21.793013 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:21.793030 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:21.793725 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:21.794101 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:21.794108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:21.794115 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:21.794371 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:21.794888 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:21.794897 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:21.795354 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 15:19:21.795715 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 15:19:21.796252 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":309,"threads_started":1}
I20260620 15:19:21.799443 13950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.124@KRBTEST.COM" created.
Jun 20 15:19:21 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260620 15:19:21.815434 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.800585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.0.97.125:34011)
0620 15:19:21.801236 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:21.801243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:21.801268 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:19:21.801307 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:21.801312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:21.801468 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:21.801599 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:21.802572 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.803577 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.805446 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.805964 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.806062 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:21.809678 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:21.809689 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:21.809700 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:21.809727 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:21.812347 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:21.813216 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:21.813223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:21.813225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:21.813386 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:21.813858 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:21.813863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:21.813868 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:21.814257 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:21.814407 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:21.814833 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 15:19:21.815165 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":502,"thread_start_us":362,"threads_started":1}
I20260620 15:19:21.815507 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.800264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34334)
0620 15:19:21.800710 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.800735 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:21.800842 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:21.801648 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:21.801655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:21.802142 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:21.802365 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:21.802377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.803727 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.803735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:21.805306 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:21.805322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:21.806116 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:21.806125 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:21.806384 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:21.807021 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:21.807047 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:21.809454 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:21.812474 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:21.812482 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:21.812556 (+ 74us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:21.813086 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:21.813514 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:21.813529 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:21.813536 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:21.813721 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:21.814386 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:21.814394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:21.814560 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 15:19:21.814705 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 15:19:21.815038 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295}
I20260620 15:19:21.818305 13950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:19:21.819609 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.819029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49344)
0620 15:19:21.819241 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.819422 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 15:19:21.819859 13899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
I20260620 15:19:21.821491 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.821112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49356)
0620 15:19:21.821304 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.821399 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 15:19:21.821686 13950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35485: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab.
I20260620 15:19:21.827633 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:35485
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:39641,127.0.97.125:34011,127.0.97.124:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:19:21.870988 13950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } attempt: 1
I20260620 15:19:21.872277 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.871645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49368)
0620 15:19:21.871902 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.872052 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 15:19:21.873450 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.872968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49374)
0620 15:19:21.873172 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.873274 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 15:19:21.873723 13950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35485: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
I20260620 15:19:21.941982 13950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } attempt: 2
I20260620 15:19:21.943130 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.942636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49384)
0620 15:19:21.942847 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.942952 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 15:19:21.944128 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.943771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49388)
0620 15:19:21.943919 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.944003 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 15:19:21.944350 13950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35485: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
I20260620 15:19:21.997519 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } attempt: 5
Jun 20 15:19:22 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260620 15:19:22.012401 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.998493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.0.97.125:34011)
0620 15:19:21.998672 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:21.998677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:21.998697 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:19:21.998878 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:21.998882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:21.998959 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:21.999043 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.000249 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.001043 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.002757 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.003190 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.003284 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.007008 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.007013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.007016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.007047 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.009718 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.010573 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.010577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.010579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.010690 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.011133 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.011137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.011139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.011392 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.011501 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.012003 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.012185 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:19:22.012765 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:21.998286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34350)
0620 15:19:21.998554 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:21.998581 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:21.998738 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:21.999134 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:21.999142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:21.999779 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:22.000059 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.000075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.001177 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.001182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.002614 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.002628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.003335 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.003342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.003511 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.004184 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:22.004207 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:22.006795 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:22.009854 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.009862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.009869 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.010447 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.010805 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.010812 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.010817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.011012 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.011520 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:22.011527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:22.011736 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 15:19:22.011925 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 15:19:22.012258 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 15:19:22.013967 13876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:19:22.015053 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.014623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49394)
0620 15:19:22.014782 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.014876 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:19:22.016029 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.015719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49402)
0620 15:19:22.015865 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.015940 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 15:19:22.016314 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35485: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
I20260620 15:19:22.035622 13950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } attempt: 3
I20260620 15:19:22.036950 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.036280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49418)
0620 15:19:22.036588 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.036720 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 15:19:22.038092 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.037643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49430)
0620 15:19:22.037876 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.037960 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260620 15:19:22.038334 13950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35485: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
I20260620 15:19:22.067528 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } attempt: 1
I20260620 15:19:22.068797 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.068300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49434)
0620 15:19:22.068550 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.068644 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 15:19:22.069773 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.069481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49442)
0620 15:19:22.069610 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.069682 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:19:22.070043 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35485: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
I20260620 15:19:22.138260 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } attempt: 2
I20260620 15:19:22.139798 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.139042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49446)
0620 15:19:22.139402 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.139529 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 15:19:22.140895 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.140599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49450)
0620 15:19:22.140724 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.140791 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:19:22.141100 13876 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35485: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
W20260620 15:19:22.150347 13957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:22.150645 13957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:22.150790 13957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:22.157831 13957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:19:22.157959 13957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:22.158058 13957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:22.158154 13957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:19:22.158224 13957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:19:22.167613 13957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:39641,127.0.97.125:34011,127.0.97.124:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:22.168946 13957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:22.170205 13957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:22.181294 13962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:22.181344 13965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:22.181596 13950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } attempt: 4
W20260620 15:19:22.182233 13963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:22.182699 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.182239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49458)
0620 15:19:22.182455 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.182546 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 15:19:22.182847 13957 server_base.cc:1034] running on GCE node
I20260620 15:19:22.183614 13957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:22.183650 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.183333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49466)
0620 15:19:22.183478 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.183552 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:19:22.183857 13950 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35485: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35485: connect: Connection refused (error 111)
I20260620 15:19:22.185045 13957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:22.186267 13957 hybrid_clock.cc:648] HybridClock initialized: now 1781968762186226 us; error 69 us; skew 500 ppm
I20260620 15:19:22.186597 13957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:22 dist-test-slave-7s64 krb5kdc[13796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:22.189572 13957 init.cc:377] Logged in from keytab as kudu/127.0.97.124@KRBTEST.COM (short username kudu)
I20260620 15:19:22.190953 13957 webserver.cc:460] Webserver started at http://127.0.97.124:33871/ using document root <none> and password file <none>
I20260620 15:19:22.191637 13957 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:22.191773 13957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:22.192073 13957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:22.194931 13957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/data/instance:
uuid: "9e44dcb74dc74514be959e543293048e"
format_stamp: "Formatted at 2026-06-20 15:19:22 on dist-test-slave-7s64"
server_key: "7a1bc7010a6a47257e0e3cd6741cb2ed"
server_key_iv: "86af0f0ad694f503b2972d2c01cd2407"
server_key_version: "encryptionkey@0"
I20260620 15:19:22.195730 13957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/wal/instance:
uuid: "9e44dcb74dc74514be959e543293048e"
format_stamp: "Formatted at 2026-06-20 15:19:22 on dist-test-slave-7s64"
server_key: "7a1bc7010a6a47257e0e3cd6741cb2ed"
server_key_iv: "86af0f0ad694f503b2972d2c01cd2407"
server_key_version: "encryptionkey@0"
I20260620 15:19:22.200700 13957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:19:22.203822 13972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:22.205103 13957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:19:22.205276 13957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/wal
uuid: "9e44dcb74dc74514be959e543293048e"
format_stamp: "Formatted at 2026-06-20 15:19:22 on dist-test-slave-7s64"
server_key: "7a1bc7010a6a47257e0e3cd6741cb2ed"
server_key_iv: "86af0f0ad694f503b2972d2c01cd2407"
server_key_version: "encryptionkey@0"
I20260620 15:19:22.205456 13957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:22.221952 13957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:22.225662 13957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:22.225920 13957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:22.243494 13957 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:35485
I20260620 15:19:22.243535 14023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:35485 every 8 connection(s)
I20260620 15:19:22.244832 13957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
I20260620 15:19:22.246368 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 13957
I20260620 15:19:22.246577 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781968566423581-389-0/minicluster-data/master-2/wal/instance
I20260620 15:19:22.247155 389 external_mini_cluster.cc:1428] Setting key 5031ed2b20406d0f542416fc5e3698c7
I20260620 15:19:22.247777 13876 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } attempt: 3
I20260620 15:19:22.250349 14024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:19:22 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260620 15:19:22.259313 14024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:19:22 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260620 15:19:22.267792 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.248301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49492)
0620 15:19:22.248550 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.248574 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:22.248692 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:22.252349 (+ 3657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:22.252354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:22.252855 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:22.253014 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.253023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.254650 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.254661 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.255570 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.255581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.256912 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.256919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.257097 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.257772 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:22.257790 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:22.260245 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:22.264962 (+ 4717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.264968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.264973 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.265325 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.265799 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.265807 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.265811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.265990 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.266781 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:22.266786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:22.266961 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 15:19:22.267108 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 15:19:22.267385 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:19:22.269035 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.249048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49492 (local address 127.0.97.124:35485)
0620 15:19:22.250178 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.250182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.250199 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:19:22.251695 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.251714 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.252047 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.252377 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.253185 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.254448 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.255715 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.256752 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.256933 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.260540 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.260550 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.260566 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.260609 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.264811 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.265455 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.265463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.265486 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.265670 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.266128 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.266133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.266137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.266628 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.266852 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.267364 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.267774 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":1022,"spinlock_wait_cycles":27904,"thread_start_us":709,"threads_started":1}
I20260620 15:19:22.270860 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.249048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49478 (local address 127.0.97.124:35485)
0620 15:19:22.249887 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.249904 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.249952 (+ 48us) server_negotiation.cc:407] Connection header received
0620 15:19:22.251722 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.251729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.252051 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.254081 (+ 2030us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.254339 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.255700 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.257458 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.258118 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.258245 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.263794 (+ 5549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.263801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.263808 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.263850 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.267699 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.268565 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.268572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.268578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.268785 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.269247 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.269253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.269255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.269593 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.269851 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.270239 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.270444 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"spinlock_wait_cycles":15232,"thread_start_us":388,"threads_started":1}
Jun 20 15:19:22 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:22.276367 13876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f: Bootstrap starting.
I20260620 15:19:22.279589 13876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:22.281086 13876 log.cc:824] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:22.283882 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.261858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.0.97.126:39641)
0620 15:19:22.262071 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.262077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.262748 (+ 671us) server_negotiation.cc:407] Connection header received
0620 15:19:22.263307 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.263314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.263386 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.263490 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.265167 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.266208 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.268409 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.270433 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.270523 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.275556 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.275566 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.275574 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.275623 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.280170 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.281310 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.281318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.281324 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.281550 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.281995 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.282000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.282003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.282424 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.282586 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.283234 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.283421 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:19:22.284765 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.261840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39641 (local address 127.0.0.1:44370)
0620 15:19:22.262606 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.262664 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:22.263166 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:22.263550 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:22.263561 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:22.264571 (+ 1010us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:22.264969 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.264982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.266445 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.266452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.268233 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.268249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.270830 (+ 2581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.270842 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.271058 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.272661 (+ 1603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:22.272684 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:22.275335 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:22.280400 (+ 5065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.280415 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.280425 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.281126 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.281658 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.281665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.281670 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.281859 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.282685 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:22.282693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:22.282865 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 15:19:22.284045 (+ 1180us) client_negotiation.cc:240] Negotiation successful
0620 15:19:22.284356 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":385,"threads_started":1}
I20260620 15:19:22.285823 13876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f: No bootstrap required, opened a new log
I20260620 15:19:22.286000 14024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:19:22 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
Jun 20 15:19:22 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260620 15:19:22.296705 13876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } }
I20260620 15:19:22.297123 13876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:22.297259 13876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1272d5a162944602a76f541a101cbb7f, State: Initialized, Role: FOLLOWER
I20260620 15:19:22.297979 13876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } }
I20260620 15:19:22.300813 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.273259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.0.97.126:39641)
0620 15:19:22.281729 (+ 8470us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.281736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.281765 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:19:22.281817 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.281824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.281913 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.282104 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.283663 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.285339 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.287378 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.288093 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.288832 (+ 739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.293069 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.293076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.293081 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.293114 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.296946 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.297742 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.297747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.297751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.297903 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.298334 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.298338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.298340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.298747 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.300082 (+ 1335us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.300193 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.300407 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":8323,"thread_start_us":8037,"threads_started":1}
I20260620 15:19:22.305013 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } } }
I20260620 15:19:22.305408 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:19:22.305496 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.287222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.0.97.125:34011)
0620 15:19:22.287378 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.287384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.287401 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:19:22.287458 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.287463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.287531 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.287617 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.288808 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.289875 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.291726 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.292382 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.292515 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.297100 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.297108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.297114 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.297158 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.301923 (+ 4765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.302958 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.302967 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.302973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.303188 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.303735 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.303743 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.303749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.304141 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.304281 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.304836 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.305040 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 15:19:22.306329 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.286662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:34011 (local address 127.0.0.1:34360)
0620 15:19:22.287007 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.287026 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:22.287111 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:22.287782 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:22.287787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:22.288403 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:22.288612 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.288625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.290083 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.290089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.291564 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.291578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.292762 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.292768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.292958 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.294034 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:22.294078 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:22.296885 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:22.302139 (+ 5254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.302147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.302153 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.302796 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.303364 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.303372 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.303378 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.303588 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.304379 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:22.304384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:22.304570 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 15:19:22.305741 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0620 15:19:22.306009 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260620 15:19:22.306447 13876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:19:22.307255 14024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:19:22 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
Jun 20 15:19:22 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260620 15:19:22.324712 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.307883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49500)
0620 15:19:22.308439 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.308544 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:22.308647 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:22.309729 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:22.309732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:22.310138 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:22.310341 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.310348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.311988 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.311993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.313382 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.313393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.314647 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.314655 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.314866 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.315487 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:22.315514 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:22.317790 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:22.321380 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.321384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.321387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.321950 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.322434 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.322441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.322447 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.322610 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.323435 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:22.323442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:22.323615 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 15:19:22.323799 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 15:19:22.324127 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":277,"threads_started":1}
I20260620 15:19:22.325708 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.309245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.0.97.124:35485)
0620 15:19:22.309438 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.309444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.309458 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:19:22.309501 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.309505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.309611 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.309711 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.310479 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.311829 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.313560 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.314440 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.314613 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.318012 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.318020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.318026 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.318070 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.321240 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.322106 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.322114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.322120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.322299 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.322748 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.322755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.322761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.323200 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.323377 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.325004 (+ 1627us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.325229 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:19:22.326247 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.302481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.0.97.125:34011)
0620 15:19:22.304524 (+ 2043us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.304530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.304553 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:19:22.304597 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.304602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.304670 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.309204 (+ 4534us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.309459 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.310361 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.311850 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.312508 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.313165 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.318519 (+ 5354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.318529 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.318550 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.318598 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.322597 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.323404 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.323409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.323412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.323581 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.324043 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.324049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.324052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.324375 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.324624 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.325645 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.325850 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1910,"thread_start_us":1730,"threads_started":1}
I20260620 15:19:22.330741 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:19:22.331204 14024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e: Bootstrap starting.
I20260620 15:19:22.324050 13809 master_runner.cc:186] Error getting master registration for 127.0.97.125:34011: OK, instance_id { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" instance_seqno: 1781968761757748 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:19:22.333170 13809 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:19:22.335153 14024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:22.336434 14024 log.cc:824] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e: Log is configured to *not* fsync() on all Append() calls
W20260620 15:19:22.343789 14045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:19:22.343916 14045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:19:22.349434 14024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e: No bootstrap required, opened a new log
I20260620 15:19:22.354748 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.336902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.0.97.126:39641)
0620 15:19:22.337082 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.337087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.337104 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:19:22.337154 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.337158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.337235 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.337327 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.339985 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.341266 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.344050 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.344660 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.344792 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.347875 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.347881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.347887 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.347926 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.351470 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.352396 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.352402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.352406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.352592 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.353050 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.353055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.353059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.353374 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.353626 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.354167 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.354387 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:19:22.360040 14024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } }
I20260620 15:19:22.360450 14024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:22.360591 14024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e44dcb74dc74514be959e543293048e, State: Initialized, Role: FOLLOWER
I20260620 15:19:22.360898 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.337108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.0.97.125:34011)
0620 15:19:22.337267 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.337274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.341068 (+ 3794us) server_negotiation.cc:407] Connection header received
0620 15:19:22.341992 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.341998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.342086 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.342154 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.346691 (+ 4537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.347738 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.349683 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.350323 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.350415 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.353836 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.353843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.353848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.353886 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.357427 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.358338 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.358344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.358347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.358487 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.358916 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.358922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.358925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.359261 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.359477 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.359983 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.360236 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:19:22.361225 14024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } }
I20260620 15:19:22.372913 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.341142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.0.97.124:35485)
0620 15:19:22.341343 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.341352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.344616 (+ 3264us) server_negotiation.cc:407] Connection header received
0620 15:19:22.351996 (+ 7380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.352005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.352092 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.352192 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.358516 (+ 6324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.359640 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.361561 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.362220 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.362316 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.366131 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.366139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.366144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.366182 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.369710 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.370521 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.370526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.370530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.370661 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.371077 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.371083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.371086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.371408 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.371725 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.372259 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.372469 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 15:19:22.373039 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } } }
I20260620 15:19:22.373432 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:19:22.374329 14024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:19:22.388978 14066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:19:22.389108 14066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:19:22.393528 14055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:19:22.393832 14055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } }
I20260620 15:19:22.395387 14055 leader_election.cc:288] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1272d5a162944602a76f541a101cbb7f (127.0.97.126:39641), 12ae3fc2f0d445ee8ddff8f8e33025a7 (127.0.97.125:34011)
I20260620 15:19:22.396921 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e44dcb74dc74514be959e543293048e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1272d5a162944602a76f541a101cbb7f" is_pre_election: true
I20260620 15:19:22.397496 13851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e44dcb74dc74514be959e543293048e in term 0.
I20260620 15:19:22.387226 13957 master_runner.cc:186] Error getting master registration for 127.0.97.125:34011: OK, instance_id { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" instance_seqno: 1781968761757748 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:19:22.398619 13957 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:19:22.399106 13974 leader_election.cc:302] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1272d5a162944602a76f541a101cbb7f, 9e44dcb74dc74514be959e543293048e; no voters:
I20260620 15:19:22.399466 14055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:19:22.399609 14055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:19:22.399713 14055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:22.403096 14055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } }
I20260620 15:19:22.404297 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e44dcb74dc74514be959e543293048e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1272d5a162944602a76f541a101cbb7f"
I20260620 15:19:22.404549 13851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:22.407939 13851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44dcb74dc74514be959e543293048e in term 1.
I20260620 15:19:22.410037 13974 leader_election.cc:302] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1272d5a162944602a76f541a101cbb7f, 9e44dcb74dc74514be959e543293048e; no voters:
I20260620 15:19:22.411700 14055 leader_election.cc:288] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [CANDIDATE]: Term 1 election: Requested vote from peers 1272d5a162944602a76f541a101cbb7f (127.0.97.126:39641), 12ae3fc2f0d445ee8ddff8f8e33025a7 (127.0.97.125:34011)
I20260620 15:19:22.411895 14055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:19:22.412026 14055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [term 1 LEADER]: Becoming Leader. State: Replica: 9e44dcb74dc74514be959e543293048e, State: Running, Role: LEADER
I20260620 15:19:22.412585 14055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } }
I20260620 15:19:22.427904 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e44dcb74dc74514be959e543293048e. Latest consensus state: current_term: 1 leader_uuid: "9e44dcb74dc74514be959e543293048e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } } }
I20260620 15:19:22.428350 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:22.429533 14070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:19:22.431658 14070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:19:22.444195 13851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [term 1 FOLLOWER]: Refusing update from remote peer 9e44dcb74dc74514be959e543293048e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:19:22.445119 14068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:19:22.469372 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e44dcb74dc74514be959e543293048e. Latest consensus state: current_term: 1 leader_uuid: "9e44dcb74dc74514be959e543293048e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } } }
I20260620 15:19:22.469702 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:19:22.473032 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e44dcb74dc74514be959e543293048e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } } }
I20260620 15:19:22.473294 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:22.483065 13950 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } attempt: 5
I20260620 15:19:22.485641 14070 catalog_manager.cc:1347] Generated new cluster ID: 45ca555594eb42e5989408d740e5021a
I20260620 15:19:22.485765 14070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 15:19:22 dist-test-slave-7s64 krb5kdc[13796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260620 15:19:22.516135 13952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.483636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35485 (local address 127.0.0.1:49508)
0620 15:19:22.484129 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:22.484150 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:22.484274 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:22.485653 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:22.485660 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:22.486200 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:22.486410 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.486423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.487998 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.488004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.489591 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:22.489607 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.490681 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.490687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:22.490878 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.491563 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:22.491584 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:22.494586 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:22.500168 (+ 5582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.500237 (+ 69us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.500286 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.502157 (+ 1871us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.503578 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:22.503587 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:22.503592 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:22.504194 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:22.506583 (+ 2389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:22.506592 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:22.507323 (+ 731us) client_negotiation.cc:769] Sending connection context
0620 15:19:22.507830 (+ 507us) client_negotiation.cc:240] Negotiation successful
0620 15:19:22.510527 (+ 2697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364}
I20260620 15:19:22.519326 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:22.484648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.0.97.124:35485)
0620 15:19:22.485059 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:22.485065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:22.485082 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:19:22.485133 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:22.485137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:22.485257 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:22.485352 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:22.486673 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.487846 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.489766 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:22.490396 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:22.490507 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:22.494821 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:22.494829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:22.494835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:22.494877 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:22.499889 (+ 5012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.502541 (+ 2652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.502551 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.502558 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.503199 (+ 641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:22.504615 (+ 1416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:22.504623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:22.504629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:22.506126 (+ 1497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:22.508947 (+ 2821us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:22.511066 (+ 2119us) server_negotiation.cc:299] Negotiation successful
0620 15:19:22.513833 (+ 2767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260620 15:19:22.526746 13950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7: Bootstrap starting.
I20260620 15:19:22.530797 13950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:22.532148 13950 log.cc:824] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:22.533116 14032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e44dcb74dc74514be959e543293048e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } } }
I20260620 15:19:22.533416 14032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:19:22.539453 13950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7: No bootstrap required, opened a new log
I20260620 15:19:22.544967 14070 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:19:22.549902 14070 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:19:22.556556 13950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } }
I20260620 15:19:22.557037 13950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:22.557202 13950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ae3fc2f0d445ee8ddff8f8e33025a7, State: Initialized, Role: FOLLOWER
I20260620 15:19:22.557940 13950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } }
I20260620 15:19:22.582034 14070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e: Generated new TSK 0
I20260620 15:19:22.588188 14070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:19:22.593205 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } } }
I20260620 15:19:22.595333 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:19:22.602986 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:19:22.603523 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } }
I20260620 15:19:22.607339 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1272d5a162944602a76f541a101cbb7f" is_pre_election: true
I20260620 15:19:22.607612 13950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:19:22.607884 13924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e44dcb74dc74514be959e543293048e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7"
I20260620 15:19:22.608292 13924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:22.609139 13925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e44dcb74dc74514be959e543293048e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" is_pre_election: true
I20260620 15:19:22.614064 13924 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e44dcb74dc74514be959e543293048e in term 1.
I20260620 15:19:22.614840 13923 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [term 1 FOLLOWER]: Refusing update from remote peer 9e44dcb74dc74514be959e543293048e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:19:22.617419 14055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:19:22.624958 14078 leader_election.cc:288] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1272d5a162944602a76f541a101cbb7f (127.0.97.126:39641), 9e44dcb74dc74514be959e543293048e (127.0.97.124:35485)
I20260620 15:19:22.625433 13999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e44dcb74dc74514be959e543293048e" is_pre_election: true
I20260620 15:19:22.626374 13899 leader_election.cc:302] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12ae3fc2f0d445ee8ddff8f8e33025a7; no voters: 1272d5a162944602a76f541a101cbb7f, 9e44dcb74dc74514be959e543293048e
I20260620 15:19:22.657598 14078 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 15:19:22.658686 14086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e44dcb74dc74514be959e543293048e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } } }
I20260620 15:19:22.659188 14086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:19:22.658650 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e44dcb74dc74514be959e543293048e. Latest consensus state: current_term: 1 leader_uuid: "9e44dcb74dc74514be959e543293048e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } } }
I20260620 15:19:22.659672 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:19:22.675362 14097 catalog_manager.cc:1259] Loaded cluster ID: 45ca555594eb42e5989408d740e5021a
I20260620 15:19:22.675534 14097 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7: loading cluster ID for follower catalog manager: success
I20260620 15:19:22.679558 14097 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7: acquiring CA information for follower catalog manager: success
I20260620 15:19:22.682130 14097 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12ae3fc2f0d445ee8ddff8f8e33025a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:19:23.053195 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e44dcb74dc74514be959e543293048e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1272d5a162944602a76f541a101cbb7f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39641 } } peers { permanent_uuid: "12ae3fc2f0d445ee8ddff8f8e33025a7" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 34011 } } peers { permanent_uuid: "9e44dcb74dc74514be959e543293048e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35485 } } }
I20260620 15:19:23.053544 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:23.345984 14045 catalog_manager.cc:1259] Loaded cluster ID: 45ca555594eb42e5989408d740e5021a
I20260620 15:19:23.346082 14045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f: loading cluster ID for follower catalog manager: success
I20260620 15:19:23.348459 14045 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f: acquiring CA information for follower catalog manager: success
I20260620 15:19:23.350175 14045 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1272d5a162944602a76f541a101cbb7f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:19:23.758759 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 13809
I20260620 15:19:23.760089 13871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:23.876104 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 13809
I20260620 15:19:23.896785 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 13883
I20260620 15:19:23.897802 13945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:23.959262 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:23.958504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39641 (local address 127.0.0.1:44388)
0620 15:19:23.958991 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:23.959102 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:39641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":254,"threads_started":1}
W20260620 15:19:23.959564 13974 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.126:39641: connect: Connection refused (error 111)
I20260620 15:19:23.961575 14103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:23.961193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39641 (local address 127.0.0.1:44394)
0620 15:19:23.961346 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:23.961411 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:39641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:19:23.961762 13974 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9e44dcb74dc74514be959e543293048e -> Peer 1272d5a162944602a76f541a101cbb7f (127.0.97.126:39641): Couldn't send request to peer 1272d5a162944602a76f541a101cbb7f. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.126:39641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 15:19:24.003726 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 13883
I20260620 15:19:24.024116 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 13957
I20260620 15:19:24.025183 14019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:24.133620 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 13957
2026-06-20T15:19:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4565 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:19:24 dist-test-slave-7s64 krb5kdc[14108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:19:24 dist-test-slave-7s64 krb5kdc[14108](info): set up 2 sockets
Jun 20 15:19:24 dist-test-slave-7s64 krb5kdc[14108](info): commencing operation
krb5kdc: starting...
W20260620 15:19:25.232925 389 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:19:25 dist-test-slave-7s64 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:19:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:19:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:19:25.435932 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44235
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:25.745266 14121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:25.745560 14121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:25.745715 14121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:25.752553 14121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:19:25.752642 14121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:25.752743 14121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:25.752823 14121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:19:25.752909 14121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:19:25.762193 14121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:25.763396 14121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:25.764695 14121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:25.775882 14126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:25.775844 14127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:25.776018 14129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:25.776703 14121 server_base.cc:1034] running on GCE node
I20260620 15:19:25.777359 14121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:25.778559 14121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:25.779752 14121 hybrid_clock.cc:648] HybridClock initialized: now 1781968765779708 us; error 30 us; skew 500 ppm
I20260620 15:19:25.780041 14121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:25 dist-test-slave-7s64 krb5kdc[14108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:25.788064 14121 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:19:25.789413 14121 webserver.cc:460] Webserver started at http://127.0.97.126:39267/ using document root <none> and password file <none>
I20260620 15:19:25.790000 14121 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:25.790103 14121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:25.790340 14121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:25.793182 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "6e1d9f08609648baab1c315dcf8dd39d"
format_stamp: "Formatted at 2026-06-20 15:19:25 on dist-test-slave-7s64"
server_key: "294d84cd6914f83ec91d6694ab318c2e"
server_key_iv: "1cdcf8b5c5a11f68e91978528ec53c02"
server_key_version: "encryptionkey@0"
I20260620 15:19:25.793812 14121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "6e1d9f08609648baab1c315dcf8dd39d"
format_stamp: "Formatted at 2026-06-20 15:19:25 on dist-test-slave-7s64"
server_key: "294d84cd6914f83ec91d6694ab318c2e"
server_key_iv: "1cdcf8b5c5a11f68e91978528ec53c02"
server_key_version: "encryptionkey@0"
I20260620 15:19:25.798616 14121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:19:25.801822 14136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:25.803126 14121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:19:25.803404 14121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "6e1d9f08609648baab1c315dcf8dd39d"
format_stamp: "Formatted at 2026-06-20 15:19:25 on dist-test-slave-7s64"
server_key: "294d84cd6914f83ec91d6694ab318c2e"
server_key_iv: "1cdcf8b5c5a11f68e91978528ec53c02"
server_key_version: "encryptionkey@0"
I20260620 15:19:25.803606 14121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:25.827855 14121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:25.831550 14121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:25.831825 14121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:25.849773 14121 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44235
I20260620 15:19:25.849809 14187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44235 every 8 connection(s)
I20260620 15:19:25.851351 14121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:19:25.853185 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 14121
I20260620 15:19:25.853390 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:19:25.854035 389 external_mini_cluster.cc:1428] Setting key 0367aee7433ed214e3374cbe811ba604
I20260620 15:19:25.855455 14188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:19:25 dist-test-slave-7s64 krb5kdc[14108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:25.869063 14188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d: Bootstrap starting.
I20260620 15:19:25.871968 14188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:25.873008 14188 log.cc:824] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:25.875717 14188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d: No bootstrap required, opened a new log
I20260620 15:19:25.876530 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:25.855790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.0.97.126:44235)
0620 15:19:25.856638 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:25.856649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:25.856687 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:19:25.858218 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:25.858237 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:25.858595 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:25.858913 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:25.860751 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:25.862071 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:25.863891 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:25.864560 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:25.864703 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:25.869325 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:25.869332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:25.869345 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:25.869381 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:25.872405 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:25.873133 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:25.873145 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:25.873169 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:25.873344 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:25.873800 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:25.873807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:25.873815 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:25.874298 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:25.874596 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:25.875053 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 15:19:25.875459 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":354,"threads_started":1}
I20260620 15:19:25.880288 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:19:25.884487 14188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1d9f08609648baab1c315dcf8dd39d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44235 } }
I20260620 15:19:25.884879 14188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:25.884984 14188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e1d9f08609648baab1c315dcf8dd39d, State: Initialized, Role: FOLLOWER
I20260620 15:19:25.885483 14188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1d9f08609648baab1c315dcf8dd39d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44235 } }
I20260620 15:19:25.885699 14188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:19:25.885828 14188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:19:25.885996 14188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:25.887995 14188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1d9f08609648baab1c315dcf8dd39d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44235 } }
I20260620 15:19:25.888447 14188 leader_election.cc:302] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e1d9f08609648baab1c315dcf8dd39d; no voters:
I20260620 15:19:25.889025 14188 leader_election.cc:288] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:19:25.889156 14193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:19:25.889843 14193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [term 1 LEADER]: Becoming Leader. State: Replica: 6e1d9f08609648baab1c315dcf8dd39d, State: Running, Role: LEADER
I20260620 15:19:25.890313 14193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1d9f08609648baab1c315dcf8dd39d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44235 } }
I20260620 15:19:25.890947 14188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:19:25.893349 14195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e1d9f08609648baab1c315dcf8dd39d. Latest consensus state: current_term: 1 leader_uuid: "6e1d9f08609648baab1c315dcf8dd39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1d9f08609648baab1c315dcf8dd39d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44235 } } }
I20260620 15:19:25.893798 14195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:25.894287 14194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e1d9f08609648baab1c315dcf8dd39d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e1d9f08609648baab1c315dcf8dd39d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44235 } } }
I20260620 15:19:25.894666 14194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:25.896462 14203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:19:25.901026 14203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:19:25.908805 14203 catalog_manager.cc:1347] Generated new cluster ID: 77326d5c69d440c5a565a0772ac5f36b
I20260620 15:19:25.908915 14203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:19:25.919072 14203 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:19:25.920229 14203 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:19:25.928755 14203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e1d9f08609648baab1c315dcf8dd39d: Generated new TSK 0
I20260620 15:19:25.929474 14203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:19:25.939657 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 14121
I20260620 15:19:25.940702 14183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:26.028228 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 14121
2026-06-20T15:19:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1886 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:19:26 dist-test-slave-7s64 krb5kdc[14216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:19:26 dist-test-slave-7s64 krb5kdc[14216](info): set up 2 sockets
Jun 20 15:19:26 dist-test-slave-7s64 krb5kdc[14216](info): commencing operation
krb5kdc: starting...
W20260620 15:19:27.112846 389 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:19:27 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:19:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:19:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:19:27.320432 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34597
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:34597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:27.635459 14229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:27.635715 14229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:27.635800 14229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:27.642774 14229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:19:27.642865 14229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:27.642910 14229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:27.642966 14229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:19:27.643002 14229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:19:27.652110 14229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:34597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:27.653205 14229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:27.654353 14229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:27.665592 14234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:27.665627 14237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:27.665649 14235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:27.666363 14229 server_base.cc:1034] running on GCE node
I20260620 15:19:27.667140 14229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:27.668394 14229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:27.669627 14229 hybrid_clock.cc:648] HybridClock initialized: now 1781968767669593 us; error 71 us; skew 500 ppm
I20260620 15:19:27.669970 14229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:27 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:27.673096 14229 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:19:27.674505 14229 webserver.cc:460] Webserver started at http://127.0.97.126:39755/ using document root <none> and password file <none>
I20260620 15:19:27.675172 14229 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:27.675328 14229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:27.675638 14229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:27.678577 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f"
format_stamp: "Formatted at 2026-06-20 15:19:27 on dist-test-slave-7s64"
server_key: "d959f9bef1d0b05564ba901b9a46acf7"
server_key_iv: "8d2807944e243662c34afc93ad1fe3dd"
server_key_version: "encryptionkey@0"
I20260620 15:19:27.679344 14229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f"
format_stamp: "Formatted at 2026-06-20 15:19:27 on dist-test-slave-7s64"
server_key: "d959f9bef1d0b05564ba901b9a46acf7"
server_key_iv: "8d2807944e243662c34afc93ad1fe3dd"
server_key_version: "encryptionkey@0"
I20260620 15:19:27.684329 14229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:19:27.687896 14244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:27.689352 14229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:19:27.689550 14229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f"
format_stamp: "Formatted at 2026-06-20 15:19:27 on dist-test-slave-7s64"
server_key: "d959f9bef1d0b05564ba901b9a46acf7"
server_key_iv: "8d2807944e243662c34afc93ad1fe3dd"
server_key_version: "encryptionkey@0"
I20260620 15:19:27.689797 14229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:27.712634 14229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:27.716367 14229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:27.716681 14229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:27.734994 14229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:34597
I20260620 15:19:27.735069 14295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:34597 every 8 connection(s)
I20260620 15:19:27.736266 14229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:19:27.737926 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 14229
I20260620 15:19:27.738157 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:19:27.738989 389 external_mini_cluster.cc:1428] Setting key f373d394dbfa9a7f4e90ba31b06c86dd
I20260620 15:19:27.740954 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:19:27 dist-test-slave-7s64 krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:27.753891 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Bootstrap starting.
I20260620 15:19:27.757949 14296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:27.759073 14296 log.cc:824] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:27.762301 14296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: No bootstrap required, opened a new log
I20260620 15:19:27.762434 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:27.740951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.0.97.126:34597)
0620 15:19:27.741763 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:27.741774 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:27.741813 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:19:27.743306 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:27.743325 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:27.743675 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:27.743991 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:27.745737 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:27.747024 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:27.748767 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:27.749360 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:27.749483 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:27.754054 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:27.754064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:27.754089 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:27.754129 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:27.757978 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:27.758728 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:27.758743 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:27.758753 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:27.758951 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:27.759354 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:27.759359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:27.759362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:27.759990 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:27.760234 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:27.760801 (+ 567us) server_negotiation.cc:299] Negotiation successful
0620 15:19:27.761196 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":364,"threads_started":1}
I20260620 15:19:27.769968 14296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:27.770371 14296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:27.770462 14296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8e5cd4a60b4d26bd2f81b0156aab0f, State: Initialized, Role: FOLLOWER
I20260620 15:19:27.771042 14296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:27.771270 14296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:19:27.771413 14296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:19:27.771600 14296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:27.773656 14296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:27.774101 14296 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8e5cd4a60b4d26bd2f81b0156aab0f; no voters:
I20260620 15:19:27.774659 14296 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:19:27.774806 14301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:19:27.775959 14301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 1 LEADER]: Becoming Leader. State: Replica: 0d8e5cd4a60b4d26bd2f81b0156aab0f, State: Running, Role: LEADER
I20260620 15:19:27.776604 14301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:27.776963 14296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:19:27.779518 14303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8e5cd4a60b4d26bd2f81b0156aab0f. Latest consensus state: current_term: 1 leader_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } } }
I20260620 15:19:27.779940 14303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:27.780547 14302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } } }
I20260620 15:19:27.780961 14302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:27.781134 14308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:19:27.785997 14308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:19:27.794997 14308 catalog_manager.cc:1347] Generated new cluster ID: cd5e0c71ed1b450899120332f6649a33
I20260620 15:19:27.795114 14308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:19:27.830242 14308 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:19:27.831537 14308 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:19:27.845463 14308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Generated new TSK 0
I20260620 15:19:27.846233 14308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:19:27.906054 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34597
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:28.218006 14322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:28.218250 14322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:28.218401 14322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:28.225333 14322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:28.225421 14322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:28.225582 14322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:19:28.234854 14322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:28.235986 14322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:28.237236 14322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:28.249752 14327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:28.249666 14330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:28.249727 14328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:28.250968 14322 server_base.cc:1034] running on GCE node
I20260620 15:19:28.251458 14322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:28.252332 14322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:28.253533 14322 hybrid_clock.cc:648] HybridClock initialized: now 1781968768253467 us; error 49 us; skew 500 ppm
I20260620 15:19:28.253782 14322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:28 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:28.256836 14322 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:19:28.258124 14322 webserver.cc:460] Webserver started at http://127.0.97.65:46515/ using document root <none> and password file <none>
I20260620 15:19:28.258733 14322 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:28.258857 14322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:28.259152 14322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:28.262005 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "e04b27eca0d640b68dab090557fe213c"
format_stamp: "Formatted at 2026-06-20 15:19:28 on dist-test-slave-7s64"
server_key: "085af855fd0917f10961b5a48bc1d4a0"
server_key_iv: "553cc50d378006fa0577b11bfe8e1865"
server_key_version: "encryptionkey@0"
I20260620 15:19:28.262735 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "e04b27eca0d640b68dab090557fe213c"
format_stamp: "Formatted at 2026-06-20 15:19:28 on dist-test-slave-7s64"
server_key: "085af855fd0917f10961b5a48bc1d4a0"
server_key_iv: "553cc50d378006fa0577b11bfe8e1865"
server_key_version: "encryptionkey@0"
I20260620 15:19:28.267452 14322 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 15:19:28.270802 14337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:28.271944 14322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:19:28.272122 14322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "e04b27eca0d640b68dab090557fe213c"
format_stamp: "Formatted at 2026-06-20 15:19:28 on dist-test-slave-7s64"
server_key: "085af855fd0917f10961b5a48bc1d4a0"
server_key_iv: "553cc50d378006fa0577b11bfe8e1865"
server_key_version: "encryptionkey@0"
I20260620 15:19:28.272302 14322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:28.303169 14322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:28.307072 14322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:28.307365 14322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:28.308326 14322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:28.309719 14322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:28.309800 14322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:28.309983 14322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:28.310041 14322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:28.342904 14322 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36825
I20260620 15:19:28.343015 14449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36825 every 8 connection(s)
I20260620 15:19:28.344182 14322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:19:28.345044 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 14322
I20260620 15:19:28.345306 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:19:28.346287 389 external_mini_cluster.cc:1428] Setting key 2270d27fd7233ddb234b9f8ea1ebfe8a
Jun 20 15:19:28 dist-test-slave-7s64 krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:28.367865 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:28.347566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:45383 (local address 127.0.97.126:34597)
0620 15:19:28.348034 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:28.348039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:28.349310 (+ 1271us) server_negotiation.cc:407] Connection header received
0620 15:19:28.351197 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:28.351205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:28.351290 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:28.351412 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:28.352957 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:28.354116 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:28.356197 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:28.356691 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:28.356800 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:28.360887 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:28.360896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:28.360904 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:28.360949 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:28.364525 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:28.365205 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:28.365213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:28.365220 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:28.365366 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:28.365782 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:28.365789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:28.365795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:28.366199 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:28.366359 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:28.367081 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 15:19:28.367343 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":242,"threads_started":1}
I20260620 15:19:28.368327 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:28.348217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:45383)
0620 15:19:28.349134 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:28.349179 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:28.350926 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:28.351622 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:28.351631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:28.352144 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:28.352785 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:28.352805 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:28.354346 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:28.354350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:28.356017 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:28.356034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:28.356919 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:28.356929 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:28.357246 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:28.357959 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:28.357992 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:28.360672 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:28.364673 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:28.364679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:28.364690 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:28.365072 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:28.365472 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:28.365475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:28.365477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:28.365658 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:28.366349 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:28.366358 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:28.366703 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 15:19:28.366915 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 15:19:28.367258 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":422,"threads_started":1}
I20260620 15:19:28.369894 14450 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34597
I20260620 15:19:28.370204 14450 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:19:28.370923 14450 heartbeater.cc:511] Master 127.0.97.126:34597 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260620 15:19:28.372944 14261 ts_manager.cc:194] Registered new tserver with Master: e04b27eca0d640b68dab090557fe213c (127.0.97.65:36825)
I20260620 15:19:28.374348 14261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:45383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:19:28.399037 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34597
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:28.711143 14456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:28.711382 14456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:28.711539 14456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:28.718609 14456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:28.718699 14456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:28.718909 14456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:19:28.729080 14456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:28.730319 14456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:28.731580 14456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:28.745893 14461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:28.745903 14462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:28.746510 14464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:28.747059 14456 server_base.cc:1034] running on GCE node
I20260620 15:19:28.747653 14456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:28.748530 14456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:28.749797 14456 hybrid_clock.cc:648] HybridClock initialized: now 1781968768749742 us; error 41 us; skew 500 ppm
I20260620 15:19:28.750160 14456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:28 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:28.753486 14456 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:19:28.754799 14456 webserver.cc:460] Webserver started at http://127.0.97.66:40863/ using document root <none> and password file <none>
I20260620 15:19:28.755398 14456 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:28.755488 14456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:28.755716 14456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:28.758430 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "b46ebe9773b2452a9e019a8600a015c9"
format_stamp: "Formatted at 2026-06-20 15:19:28 on dist-test-slave-7s64"
server_key: "cce581704c8b02246d6efc179afe9433"
server_key_iv: "4d9141976fc7923e4c622861871e00c8"
server_key_version: "encryptionkey@0"
I20260620 15:19:28.759128 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "b46ebe9773b2452a9e019a8600a015c9"
format_stamp: "Formatted at 2026-06-20 15:19:28 on dist-test-slave-7s64"
server_key: "cce581704c8b02246d6efc179afe9433"
server_key_iv: "4d9141976fc7923e4c622861871e00c8"
server_key_version: "encryptionkey@0"
I20260620 15:19:28.767360 14456 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260620 15:19:28.772676 14471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:28.774677 14456 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:19:28.775054 14456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "b46ebe9773b2452a9e019a8600a015c9"
format_stamp: "Formatted at 2026-06-20 15:19:28 on dist-test-slave-7s64"
server_key: "cce581704c8b02246d6efc179afe9433"
server_key_iv: "4d9141976fc7923e4c622861871e00c8"
server_key_version: "encryptionkey@0"
I20260620 15:19:28.775305 14456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:28.797827 14456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:28.803334 14456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:28.803628 14456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:28.804688 14456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:28.805996 14456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:28.806106 14456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:28.806267 14456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:28.806339 14456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:28.838577 14456 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:43137
I20260620 15:19:28.838654 14583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:43137 every 8 connection(s)
I20260620 15:19:28.839783 14456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:19:28.847415 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 14456
I20260620 15:19:28.847617 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:19:28.848062 389 external_mini_cluster.cc:1428] Setting key e6cfab5a66a1280e4744d63db0d4be19
Jun 20 15:19:28 dist-test-slave-7s64 krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:28.859254 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:28.842291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:33025 (local address 127.0.97.126:34597)
0620 15:19:28.842468 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:28.842472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:28.843657 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 15:19:28.845505 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:28.845510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:28.845605 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:28.845715 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:28.847124 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:28.847914 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:28.849486 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:28.849910 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:28.850046 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:28.853636 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:28.853641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:28.853644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:28.853676 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:28.856262 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:28.857040 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:28.857046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:28.857050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:28.857193 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:28.857626 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:28.857638 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:28.857640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:28.857889 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:28.858044 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:28.858681 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 15:19:28.858926 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:19:28.860172 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:28.842792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:33025)
0620 15:19:28.843495 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:28.843536 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:28.845248 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:28.845939 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:28.845947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:28.846416 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:28.846927 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:28.846942 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:28.848047 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:28.848062 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:28.849345 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:28.849357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:28.850140 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:28.850147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:28.850358 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:28.851015 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:28.851041 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:28.853383 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:28.856413 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:28.856422 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:28.856436 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:28.856914 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:28.857318 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:28.857323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:28.857327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:28.857506 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:28.858012 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:28.858019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:28.858333 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 15:19:28.858586 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 15:19:28.859067 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":295,"threads_started":1}
I20260620 15:19:28.861598 14584 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34597
I20260620 15:19:28.861924 14584 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:28.862641 14584 heartbeater.cc:511] Master 127.0.97.126:34597 requested a full tablet report, sending...
I20260620 15:19:28.864076 14261 ts_manager.cc:194] Registered new tserver with Master: b46ebe9773b2452a9e019a8600a015c9 (127.0.97.66:43137)
I20260620 15:19:28.865366 14261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:33025
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:19:28.896390 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34597
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:29.211983 14589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:29.212232 14589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:29.212409 14589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:29.219249 14589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:29.219365 14589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:29.219549 14589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:19:29.228468 14589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:29.229899 14589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:29.231105 14589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:29.243323 14594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:29.243285 14595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:29.243374 14597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:29.244091 14589 server_base.cc:1034] running on GCE node
I20260620 15:19:29.244755 14589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:29.245836 14589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:29.247035 14589 hybrid_clock.cc:648] HybridClock initialized: now 1781968769247001 us; error 28 us; skew 500 ppm
I20260620 15:19:29.247334 14589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:29 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:29.250391 14589 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:19:29.251677 14589 webserver.cc:460] Webserver started at http://127.0.97.67:43237/ using document root <none> and password file <none>
I20260620 15:19:29.252287 14589 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:29.252430 14589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:29.252754 14589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:29.255458 14589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "9020128448114a8fab0c511359d980cf"
format_stamp: "Formatted at 2026-06-20 15:19:29 on dist-test-slave-7s64"
server_key: "ae4346d44bf4e2326c893dc8f13040bb"
server_key_iv: "240e1e683461c1b65150409e0e34099b"
server_key_version: "encryptionkey@0"
I20260620 15:19:29.256207 14589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "9020128448114a8fab0c511359d980cf"
format_stamp: "Formatted at 2026-06-20 15:19:29 on dist-test-slave-7s64"
server_key: "ae4346d44bf4e2326c893dc8f13040bb"
server_key_iv: "240e1e683461c1b65150409e0e34099b"
server_key_version: "encryptionkey@0"
I20260620 15:19:29.261047 14589 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:19:29.264467 14604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:29.265775 14589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:19:29.265936 14589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "9020128448114a8fab0c511359d980cf"
format_stamp: "Formatted at 2026-06-20 15:19:29 on dist-test-slave-7s64"
server_key: "ae4346d44bf4e2326c893dc8f13040bb"
server_key_iv: "240e1e683461c1b65150409e0e34099b"
server_key_version: "encryptionkey@0"
I20260620 15:19:29.266124 14589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:29.298341 14589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:29.302407 14589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:29.302686 14589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:29.303541 14589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:29.304929 14589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:29.305027 14589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:29.305179 14589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:29.305249 14589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:29.336539 14589 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:46641
I20260620 15:19:29.336619 14716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:46641 every 8 connection(s)
I20260620 15:19:29.337733 14589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:19:29.345350 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 14589
I20260620 15:19:29.345534 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:19:29.345963 389 external_mini_cluster.cc:1428] Setting key 84696cfe61dec81846a317e2db1a6a91
Jun 20 15:19:29 dist-test-slave-7s64 krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:29.357125 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.340146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:58801 (local address 127.0.97.126:34597)
0620 15:19:29.340383 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:29.340390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:29.341638 (+ 1248us) server_negotiation.cc:407] Connection header received
0620 15:19:29.343433 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:29.343440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:29.343520 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:29.343635 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:29.345100 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.346264 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.348252 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.348734 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.348872 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.352069 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:29.352073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:29.352076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:29.352116 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:29.354643 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:29.355264 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:29.355268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:29.355270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:29.355378 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:29.355732 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:29.355735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:29.355737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:29.355973 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:29.356089 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:29.356725 (+ 636us) server_negotiation.cc:299] Negotiation successful
0620 15:19:29.356881 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 15:19:29.357996 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.340771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:58801)
0620 15:19:29.341472 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:29.341510 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:29.343176 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:29.343835 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:29.343842 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:29.344301 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:29.344897 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.344914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.346431 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.346438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.348113 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.348125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.348952 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.348961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.349211 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.349791 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:29.349816 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:29.351896 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:29.354761 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:29.354767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:29.354778 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:29.355171 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:29.355482 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:29.355485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:29.355487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:29.355644 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:29.356079 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:29.356084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:29.356356 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 15:19:29.356643 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 15:19:29.356990 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":269,"threads_started":1}
I20260620 15:19:29.359280 14717 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34597
I20260620 15:19:29.359570 14717 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:29.360353 14717 heartbeater.cc:511] Master 127.0.97.126:34597 requested a full tablet report, sending...
I20260620 15:19:29.361887 14261 ts_manager.cc:194] Registered new tserver with Master: 9020128448114a8fab0c511359d980cf (127.0.97.67:46641)
I20260620 15:19:29.362676 14261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:58801
I20260620 15:19:29.373320 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:19:29.377208 14450 heartbeater.cc:503] Master 127.0.97.126:34597 was elected leader, sending a full tablet report...
I20260620 15:19:29.389472 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.377793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59702 (local address 127.0.97.126:34597)
0620 15:19:29.377974 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:29.377980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:29.378396 (+ 416us) server_negotiation.cc:407] Connection header received
0620 15:19:29.378583 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:29.378588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:29.378657 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:29.378770 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:29.380060 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.380893 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.382227 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.382631 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.382721 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.384933 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:29.384937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:29.384941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:29.384969 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:29.387435 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:29.387975 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:29.387979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:29.387981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:29.388089 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:29.388376 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:29.388379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:29.388381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:29.388725 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:29.388901 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:29.389119 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 15:19:29.389262 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:19:29.392969 14261 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:19:29.399562 14261 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:19:29.441243 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.410210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.0.1:50926)
0620 15:19:29.429393 (+ 19183us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:29.429444 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:29.429641 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:29.433163 (+ 3522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:29.433178 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:29.433222 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:29.434458 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.434475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.436228 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.436234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.438672 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.438686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.440243 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.440249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.440431 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.440516 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 15:19:29.440658 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 15:19:29.440842 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18870,"thread_start_us":18690,"threads_started":1}
I20260620 15:19:29.442188 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.410480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.0.97.65:36825)
0620 15:19:29.432376 (+ 21896us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:29.432387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:29.432419 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:19:29.432538 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:29.432546 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:29.432825 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:29.433022 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:29.434677 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.436093 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.438833 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.440047 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.440193 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.440441 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:29.440705 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 15:19:29.441882 (+ 1177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21718,"thread_start_us":21562,"threads_started":1}
I20260620 15:19:29.442384 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.411141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.0.97.66:43137)
0620 15:19:29.433122 (+ 21981us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:29.433129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:29.434695 (+ 1566us) server_negotiation.cc:407] Connection header received
0620 15:19:29.434808 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:29.434817 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:29.435021 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:29.435222 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:29.435847 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.437204 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.439632 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.441045 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.441182 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.441373 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:29.441898 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 15:19:29.442067 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21815,"thread_start_us":21722,"threads_started":1}
I20260620 15:19:29.442540 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.410690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:46641 (local address 127.0.0.1:48314)
0620 15:19:29.433990 (+ 23300us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:29.434007 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:29.434220 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:29.434837 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:29.434841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:29.434865 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:29.435277 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.435288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.437191 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.437196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.439220 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.439229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.441242 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.441248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.441690 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.441737 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 15:19:29.441828 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 15:19:29.441961 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22871,"mutex_wait_us":54,"thread_start_us":23262,"threads_started":1}
I20260620 15:19:29.443054 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.411597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.0.97.67:46641)
0620 15:19:29.432716 (+ 21119us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:29.432723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:29.434148 (+ 1425us) server_negotiation.cc:407] Connection header received
0620 15:19:29.434423 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:29.434431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:29.434652 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:29.434853 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:29.435515 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.436991 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.439357 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.440754 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.440878 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.441077 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:29.442632 (+ 1555us) server_negotiation.cc:299] Negotiation successful
0620 15:19:29.442798 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20968,"thread_start_us":20818,"threads_started":1}
I20260620 15:19:29.443213 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.410990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43137 (local address 127.0.0.1:46312)
0620 15:19:29.434449 (+ 23459us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:29.434466 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:29.434587 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:29.435229 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:29.435239 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:29.435268 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:29.435684 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.435696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.437363 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.437369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.439470 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.439486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.441505 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.441511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.441684 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.441747 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 15:19:29.441825 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 15:19:29.441977 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23302,"mutex_wait_us":46,"thread_start_us":22286,"threads_started":1}
I20260620 15:19:29.445851 14385 tablet_service.cc:1467] Processing CreateTablet for tablet a8a3543b4e674ee0b986d24427f148d4 (DEFAULT_TABLE table=test-table [id=783ecbe024e94106aeb13fbd84913e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:29.446193 14652 tablet_service.cc:1467] Processing CreateTablet for tablet a8a3543b4e674ee0b986d24427f148d4 (DEFAULT_TABLE table=test-table [id=783ecbe024e94106aeb13fbd84913e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:29.447057 14385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a3543b4e674ee0b986d24427f148d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:29.447353 14652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a3543b4e674ee0b986d24427f148d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:29.452862 14519 tablet_service.cc:1467] Processing CreateTablet for tablet a8a3543b4e674ee0b986d24427f148d4 (DEFAULT_TABLE table=test-table [id=783ecbe024e94106aeb13fbd84913e47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:29.454203 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a3543b4e674ee0b986d24427f148d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:29.457796 14735 tablet_bootstrap.cc:492] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: Bootstrap starting.
I20260620 15:19:29.460804 14735 tablet_bootstrap.cc:654] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:29.461789 14735 log.cc:824] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:29.464824 14735 tablet_bootstrap.cc:492] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: No bootstrap required, opened a new log
I20260620 15:19:29.465095 14735 ts_tablet_manager.cc:1397] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 15:19:29.465334 14736 tablet_bootstrap.cc:492] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: Bootstrap starting.
I20260620 15:19:29.468607 14736 tablet_bootstrap.cc:654] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:29.468726 14738 tablet_bootstrap.cc:492] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: Bootstrap starting.
I20260620 15:19:29.469893 14736 log.cc:824] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:29.472172 14738 tablet_bootstrap.cc:654] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:29.473183 14738 log.cc:824] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:29.473356 14736 tablet_bootstrap.cc:492] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: No bootstrap required, opened a new log
I20260620 15:19:29.473611 14736 ts_tablet_manager.cc:1397] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260620 15:19:29.476054 14738 tablet_bootstrap.cc:492] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: No bootstrap required, opened a new log
I20260620 15:19:29.475842 14735 raft_consensus.cc:348] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } }
I20260620 15:19:29.476253 14735 raft_consensus.cc:374] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:29.476338 14738 ts_tablet_manager.cc:1397] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:19:29.476352 14735 raft_consensus.cc:729] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04b27eca0d640b68dab090557fe213c, State: Initialized, Role: FOLLOWER
I20260620 15:19:29.477017 14735 consensus_queue.cc:260] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } }
I20260620 15:19:29.478672 14735 ts_tablet_manager.cc:1428] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 15:19:29.483668 14736 raft_consensus.cc:348] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } }
I20260620 15:19:29.484076 14736 raft_consensus.cc:374] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:29.484205 14736 raft_consensus.cc:729] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9020128448114a8fab0c511359d980cf, State: Initialized, Role: FOLLOWER
I20260620 15:19:29.484779 14736 consensus_queue.cc:260] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } }
I20260620 15:19:29.485819 14717 heartbeater.cc:503] Master 127.0.97.126:34597 was elected leader, sending a full tablet report...
I20260620 15:19:29.485764 14738 raft_consensus.cc:348] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } }
I20260620 15:19:29.486187 14738 raft_consensus.cc:374] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:29.486285 14738 raft_consensus.cc:729] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46ebe9773b2452a9e019a8600a015c9, State: Initialized, Role: FOLLOWER
I20260620 15:19:29.486367 14736 ts_tablet_manager.cc:1428] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 15:19:29.486804 14738 consensus_queue.cc:260] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } }
I20260620 15:19:29.487916 14584 heartbeater.cc:503] Master 127.0.97.126:34597 was elected leader, sending a full tablet report...
I20260620 15:19:29.488345 14738 ts_tablet_manager.cc:1428] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260620 15:19:29.589337 14718 tablet.cc:2369] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:19:29.591641 14585 tablet.cc:2369] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:19:29.597347 14451 tablet.cc:2369] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:19:29.672708 14743 raft_consensus.cc:482] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:19:29.672951 14743 raft_consensus.cc:504] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } }
I20260620 15:19:29.674516 14743 leader_election.cc:288] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e04b27eca0d640b68dab090557fe213c (127.0.97.65:36825), 9020128448114a8fab0c511359d980cf (127.0.97.67:46641)
I20260620 15:19:29.683907 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.675009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:40931 (local address 127.0.97.65:36825)
0620 15:19:29.675225 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:29.675231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:29.676612 (+ 1381us) server_negotiation.cc:407] Connection header received
0620 15:19:29.676793 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:29.676801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:29.676883 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:29.676982 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:29.677694 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.678712 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.681566 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.682757 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.682883 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.682979 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:29.683535 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 15:19:29.683658 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:19:29.683907 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.674909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44539 (local address 127.0.97.67:46641)
0620 15:19:29.675127 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:29.675135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:29.676295 (+ 1160us) server_negotiation.cc:407] Connection header received
0620 15:19:29.676547 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:29.676554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:29.676669 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:29.676819 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:29.677505 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.678610 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.681585 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.682756 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.682883 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.682979 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:29.683535 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 15:19:29.683679 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:19:29.684140 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.675405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:46641 (local address 127.0.97.66:44539)
0620 15:19:29.676171 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:29.676194 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:29.676364 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:29.676874 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:29.676877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:29.676897 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:29.677361 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.677375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.678729 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.678732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.681401 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.681412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.683017 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.683021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.683330 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.683405 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 15:19:29.683496 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 15:19:29.683662 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":256,"thread_start_us":427,"threads_started":1}
I20260620 15:19:29.684166 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.675467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.97.66:40931)
0620 15:19:29.676489 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:29.676523 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:29.676653 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:29.677028 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:29.677033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:29.677123 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:29.677543 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.677555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.678834 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.678843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.681398 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:29.681412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.683012 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.683018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:29.683326 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.683400 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 15:19:29.683496 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:19:29.683662 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":261,"spinlock_wait_cycles":23808,"thread_start_us":552,"threads_started":1}
I20260620 15:19:29.685271 14672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3543b4e674ee0b986d24427f148d4" candidate_uuid: "b46ebe9773b2452a9e019a8600a015c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9020128448114a8fab0c511359d980cf" is_pre_election: true
I20260620 15:19:29.685273 14405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3543b4e674ee0b986d24427f148d4" candidate_uuid: "b46ebe9773b2452a9e019a8600a015c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04b27eca0d640b68dab090557fe213c" is_pre_election: true
I20260620 15:19:29.685773 14672 raft_consensus.cc:2454] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b46ebe9773b2452a9e019a8600a015c9 in term 0.
I20260620 15:19:29.685771 14405 raft_consensus.cc:2454] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b46ebe9773b2452a9e019a8600a015c9 in term 0.
I20260620 15:19:29.686406 14472 leader_election.cc:302] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b46ebe9773b2452a9e019a8600a015c9, e04b27eca0d640b68dab090557fe213c; no voters:
I20260620 15:19:29.686827 14743 raft_consensus.cc:2781] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:19:29.686966 14743 raft_consensus.cc:482] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:19:29.687080 14743 raft_consensus.cc:3037] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:29.689471 14743 raft_consensus.cc:504] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } }
I20260620 15:19:29.690105 14743 leader_election.cc:288] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [CANDIDATE]: Term 1 election: Requested vote from peers e04b27eca0d640b68dab090557fe213c (127.0.97.65:36825), 9020128448114a8fab0c511359d980cf (127.0.97.67:46641)
I20260620 15:19:29.690478 14405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3543b4e674ee0b986d24427f148d4" candidate_uuid: "b46ebe9773b2452a9e019a8600a015c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04b27eca0d640b68dab090557fe213c"
I20260620 15:19:29.690604 14672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3543b4e674ee0b986d24427f148d4" candidate_uuid: "b46ebe9773b2452a9e019a8600a015c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9020128448114a8fab0c511359d980cf"
I20260620 15:19:29.690676 14405 raft_consensus.cc:3037] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:29.690821 14672 raft_consensus.cc:3037] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:29.693849 14405 raft_consensus.cc:2454] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b46ebe9773b2452a9e019a8600a015c9 in term 1.
I20260620 15:19:29.693890 14672 raft_consensus.cc:2454] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b46ebe9773b2452a9e019a8600a015c9 in term 1.
I20260620 15:19:29.694257 14473 leader_election.cc:302] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9020128448114a8fab0c511359d980cf, b46ebe9773b2452a9e019a8600a015c9; no voters:
I20260620 15:19:29.694587 14743 raft_consensus.cc:2781] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:19:29.695029 14743 raft_consensus.cc:686] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 1 LEADER]: Becoming Leader. State: Replica: b46ebe9773b2452a9e019a8600a015c9, State: Running, Role: LEADER
I20260620 15:19:29.695513 14743 consensus_queue.cc:237] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } }
I20260620 15:19:29.699698 14261 catalog_manager.cc:5495] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b46ebe9773b2452a9e019a8600a015c9 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "b46ebe9773b2452a9e019a8600a015c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } health_report { overall_health: HEALTHY } } }
I20260620 15:19:29.733722 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.728990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.0.97.66:43137)
0620 15:19:29.729156 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:29.729160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:29.729175 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:19:29.729302 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:29.729305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:29.729382 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:29.729493 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:19:29.729814 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.730495 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.731816 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.732205 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.732288 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.732716 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:19:29.733177 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:19:29.733292 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:29.733459 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 15:19:29.733519 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:19:29.741981 14405 raft_consensus.cc:1261] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c [term 1 FOLLOWER]: Refusing update from remote peer b46ebe9773b2452a9e019a8600a015c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:19:29.742127 14672 raft_consensus.cc:1261] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf [term 1 FOLLOWER]: Refusing update from remote peer b46ebe9773b2452a9e019a8600a015c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:19:29.742980 14746 consensus_queue.cc:1035] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:19:29.743386 14743 consensus_queue.cc:1035] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:19:29.754809 14749 mvcc.cc:205] Tried to move back new op lower bound from 7298944080853114880 to 7298944080676294656. Current Snapshot: MvccSnapshot[applied={T|T < 7298944080853114880}]
I20260620 15:19:29.760314 14750 mvcc.cc:205] Tried to move back new op lower bound from 7298944080853114880 to 7298944080676294656. Current Snapshot: MvccSnapshot[applied={T|T < 7298944080853114880}]
I20260620 15:19:29.770037 14260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:19:29.770334 14260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:19:29.774410 14260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Sending DeleteTablet for 3 replicas of tablet a8a3543b4e674ee0b986d24427f148d4
I20260620 15:19:29.775408 14385 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a3543b4e674ee0b986d24427f148d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:29 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:50926
I20260620 15:19:29.775632 14652 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a3543b4e674ee0b986d24427f148d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:29 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:48314
I20260620 15:19:29.775935 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a3543b4e674ee0b986d24427f148d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:29 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:46312
I20260620 15:19:29.776141 14651 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a3543b4e674ee0b986d24427f148d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:29 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:48314
I20260620 15:19:29.777679 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 14322
I20260620 15:19:29.785543 14761 tablet_replica.cc:330] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: stopping tablet replica
I20260620 15:19:29.787807 14761 raft_consensus.cc:2229] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:19:29.788414 14761 raft_consensus.cc:2258] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:29.792150 14761 ts_tablet_manager.cc:1905] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:29.794852 14760 tablet_replica.cc:330] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: stopping tablet replica
I20260620 15:19:29.795226 14760 raft_consensus.cc:2229] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:19:29.795675 14760 raft_consensus.cc:2258] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:29.796283 14762 tablet_replica.cc:330] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: stopping tablet replica
I20260620 15:19:29.796671 14762 raft_consensus.cc:2229] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:19:29.797277 14762 raft_consensus.cc:2258] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 15:19:29 dist-test-slave-7s64 krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:19:29.799353 14760 ts_tablet_manager.cc:1905] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:29.800640 14761 ts_tablet_manager.cc:1918] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:29.800796 14761 log.cc:1196] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/a8a3543b4e674ee0b986d24427f148d4
I20260620 15:19:29.801343 14761 ts_tablet_manager.cc:1939] T a8a3543b4e674ee0b986d24427f148d4 P 9020128448114a8fab0c511359d980cf: Deleting consensus metadata
I20260620 15:19:29.802547 14762 ts_tablet_manager.cc:1905] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:29.804312 14246 catalog_manager.cc:4841] TS 9020128448114a8fab0c511359d980cf (127.0.97.67:46641): tablet a8a3543b4e674ee0b986d24427f148d4 (table test-table [id=783ecbe024e94106aeb13fbd84913e47]) successfully deleted
W20260620 15:19:29.804814 14246 catalog_manager.cc:4808] TS 9020128448114a8fab0c511359d980cf (127.0.97.67:46641): delete failed for tablet a8a3543b4e674ee0b986d24427f148d4 because the tablet was not found. No further retry: Not found: Tablet not found: a8a3543b4e674ee0b986d24427f148d4
I20260620 15:19:29.808848 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.778863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.0.97.65:36825)
0620 15:19:29.779028 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:29.779034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:29.787138 (+ 8104us) server_negotiation.cc:407] Connection header received
0620 15:19:29.787370 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:29.787375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:29.787490 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:29.787596 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:29.789669 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.791020 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.793192 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.793875 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.793972 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.799920 (+ 5948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:29.799934 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:29.799951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:29.799998 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:29.803820 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:29.806082 (+ 2262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:29.806091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:29.806095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:29.806290 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:29.806797 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:29.806802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:29.806806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:29.807280 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:29.807581 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:29.808191 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 15:19:29.808410 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:19:29.809063 14760 ts_tablet_manager.cc:1918] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:29.809211 14760 log.cc:1196] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/a8a3543b4e674ee0b986d24427f148d4
I20260620 15:19:29.809654 14762 ts_tablet_manager.cc:1918] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:29.809700 14760 ts_tablet_manager.cc:1939] T a8a3543b4e674ee0b986d24427f148d4 P e04b27eca0d640b68dab090557fe213c: Deleting consensus metadata
I20260620 15:19:29.809816 14762 log.cc:1196] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/a8a3543b4e674ee0b986d24427f148d4
I20260620 15:19:29.810320 14762 ts_tablet_manager.cc:1939] T a8a3543b4e674ee0b986d24427f148d4 P b46ebe9773b2452a9e019a8600a015c9: Deleting consensus metadata
I20260620 15:19:29.810699 14445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:29.811414 14245 catalog_manager.cc:4841] TS e04b27eca0d640b68dab090557fe213c (127.0.97.65:36825): tablet a8a3543b4e674ee0b986d24427f148d4 (table test-table [id=783ecbe024e94106aeb13fbd84913e47]) successfully deleted
I20260620 15:19:29.811800 14248 catalog_manager.cc:4841] TS b46ebe9773b2452a9e019a8600a015c9 (127.0.97.66:43137): tablet a8a3543b4e674ee0b986d24427f148d4 (table test-table [id=783ecbe024e94106aeb13fbd84913e47]) successfully deleted
I20260620 15:19:29.918331 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 14322
I20260620 15:19:29.942104 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 14456
Jun 20 15:19:29 dist-test-slave-7s64 krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:19:29.959149 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:29.942834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.0.97.66:43137)
0620 15:19:29.943033 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:29.943039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:29.943060 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:19:29.943294 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:29.943300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:29.943389 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:29.943501 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:29.944830 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.946062 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.947308 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:29.947975 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:29.948102 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:29.952352 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:29.952367 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:29.952385 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:29.952422 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:29.956176 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:29.957044 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:29.957053 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:29.957057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:29.957218 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:29.957577 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:29.957582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:29.957586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:29.958046 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:29.958385 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:29.958536 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 15:19:29.958750 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:19:29.959859 14579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:30.071487 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 14456
I20260620 15:19:30.090391 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 14589
Jun 20 15:19:30 dist-test-slave-7s64 krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:19:30.105129 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:30.091231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.0.97.67:46641)
0620 15:19:30.091393 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:30.091397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:30.091414 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:19:30.091599 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:30.091602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:30.091666 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:30.091744 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:30.093069 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:30.093965 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:30.095183 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:30.095605 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:30.095710 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:30.099867 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:30.099881 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:30.099895 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:30.099931 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:30.102748 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:30.103348 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:30.103355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:30.103357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:30.103491 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:30.103783 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:30.103786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:30.103788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:30.104113 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:30.104322 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:30.104629 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 15:19:30.104784 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:19:30.105808 14712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:30.194850 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 14589
I20260620 15:19:30.211459 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 14229
I20260620 15:19:30.212682 14291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:30.321031 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 14229
I20260620 15:19:30.341641 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34597
--webserver_interface=127.0.97.126
--webserver_port=39755
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:34597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:30.654209 14768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:30.654489 14768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:30.654667 14768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:30.661906 14768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:19:30.662004 14768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:30.662094 14768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:30.662216 14768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:19:30.662302 14768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:19:30.671933 14768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:34597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:30.673254 14768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:30.674504 14768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:30.696875 14774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:30.696903 14775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:30.696857 14777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:30.697813 14768 server_base.cc:1034] running on GCE node
I20260620 15:19:30.698668 14768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:30.700191 14768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:30.701442 14768 hybrid_clock.cc:648] HybridClock initialized: now 1781968770701387 us; error 67 us; skew 500 ppm
I20260620 15:19:30.701844 14768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:30 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:30.705407 14768 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:19:30.706984 14768 webserver.cc:460] Webserver started at http://127.0.97.126:39755/ using document root <none> and password file <none>
I20260620 15:19:30.707744 14768 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:30.707888 14768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:30.713733 14768 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:19:30.716658 14784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:30.717955 14768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:19:30.718144 14768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f"
format_stamp: "Formatted at 2026-06-20 15:19:27 on dist-test-slave-7s64"
server_key: "d959f9bef1d0b05564ba901b9a46acf7"
server_key_iv: "8d2807944e243662c34afc93ad1fe3dd"
server_key_version: "encryptionkey@0"
I20260620 15:19:30.719367 14768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:30.731244 14768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:30.735491 14768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:30.735771 14768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:30.754971 14768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:34597
I20260620 15:19:30.754987 14835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:34597 every 8 connection(s)
I20260620 15:19:30.756306 14768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:19:30.761183 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 14768
I20260620 15:19:30.762141 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:36825
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:34597
--webserver_port=46515
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:19:30.763774 14836 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:19:30.769006 14836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Bootstrap starting.
I20260620 15:19:30.788606 14836 log.cc:824] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:30.800211 14836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:19:30.800819 14836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Bootstrap complete.
I20260620 15:19:30.809886 14836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:30.810679 14836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8e5cd4a60b4d26bd2f81b0156aab0f, State: Initialized, Role: FOLLOWER
I20260620 15:19:30.811314 14836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:30.811547 14836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:19:30.811662 14836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:19:30.811851 14836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:19:30.815448 14836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:30.815944 14836 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8e5cd4a60b4d26bd2f81b0156aab0f; no voters:
I20260620 15:19:30.816602 14836 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:19:30.816746 14840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:19:30.818997 14836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:19:30.823402 14840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 2 LEADER]: Becoming Leader. State: Replica: 0d8e5cd4a60b4d26bd2f81b0156aab0f, State: Running, Role: LEADER
I20260620 15:19:30.823941 14840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:30.826498 14842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8e5cd4a60b4d26bd2f81b0156aab0f. Latest consensus state: current_term: 2 leader_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } } }
I20260620 15:19:30.826463 14841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } } }
I20260620 15:19:30.826905 14841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:30.826906 14842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:30.836150 14851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:19:30.844049 14851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:19:30.845721 14851 catalog_manager.cc:1259] Loaded cluster ID: cd5e0c71ed1b450899120332f6649a33
I20260620 15:19:30.845835 14851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:19:30.848943 14851 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 15:19:30.865995 14851 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 15:19:30.875003 14851 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:19:30.876560 14851 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Loaded TSK: 0
I20260620 15:19:30.877975 14851 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:19:31.105659 14838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:31.105909 14838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:31.106053 14838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:31.113583 14838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:31.113701 14838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:31.113889 14838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:19:31.124476 14838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:36825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=46515
--tserver_master_addrs=127.0.97.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:31.125774 14838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:31.127122 14838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:31.143872 14865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:31.144438 14864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:31.143841 14867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:31.145042 14838 server_base.cc:1034] running on GCE node
I20260620 15:19:31.145570 14838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:31.146410 14838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:31.147616 14838 hybrid_clock.cc:648] HybridClock initialized: now 1781968771147565 us; error 54 us; skew 500 ppm
I20260620 15:19:31.147919 14838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:31 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:31.155656 14838 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:19:31.157260 14838 webserver.cc:460] Webserver started at http://127.0.97.65:46515/ using document root <none> and password file <none>
I20260620 15:19:31.157928 14838 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:31.158066 14838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:31.163367 14838 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 15:19:31.166181 14874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:31.167569 14838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:19:31.167718 14838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "e04b27eca0d640b68dab090557fe213c"
format_stamp: "Formatted at 2026-06-20 15:19:28 on dist-test-slave-7s64"
server_key: "085af855fd0917f10961b5a48bc1d4a0"
server_key_iv: "553cc50d378006fa0577b11bfe8e1865"
server_key_version: "encryptionkey@0"
I20260620 15:19:31.168691 14838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:31.206616 14838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:31.211436 14838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:31.211717 14838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:31.212749 14838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:31.214008 14838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:31.214125 14838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:31.214252 14838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:31.214352 14838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:31.247007 14838 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36825
I20260620 15:19:31.247113 14986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36825 every 8 connection(s)
I20260620 15:19:31.248370 14838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:19:31.256266 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 14838
I20260620 15:19:31.257256 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:43137
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:34597
--webserver_port=40863
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:19:31 dist-test-slave-7s64 krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:31.273797 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:31.251246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57681 (local address 127.0.97.126:34597)
0620 15:19:31.251836 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:31.251853 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:31.252529 (+ 676us) server_negotiation.cc:407] Connection header received
0620 15:19:31.254486 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:31.254502 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:31.254819 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:31.255131 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:31.256523 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:31.258672 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:31.260277 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:31.261423 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:31.261632 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:31.266203 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:31.266210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:31.266222 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:31.266256 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:31.269186 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:31.270189 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:31.270195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:31.270199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:31.270341 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:31.270863 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:31.270866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:31.270868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:31.271261 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:31.271398 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:31.272385 (+ 987us) server_negotiation.cc:299] Negotiation successful
0620 15:19:31.272790 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":232,"threads_started":1}
I20260620 15:19:31.274225 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:31.251584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:57681)
0620 15:19:31.252347 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:31.252383 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:31.253998 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:31.255244 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:31.255251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:31.255716 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:31.256240 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:31.256255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:31.258852 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:31.258856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:31.260109 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:31.260122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:31.261671 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:31.261681 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:31.261993 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:31.262782 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:31.262816 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:31.265961 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:31.269337 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:31.269349 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:31.269365 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:31.270051 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:31.270479 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:31.270498 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:31.270505 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:31.270746 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:31.271424 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:31.271435 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:31.271890 (+ 455us) client_negotiation.cc:769] Sending connection context
0620 15:19:31.272227 (+ 337us) client_negotiation.cc:240] Negotiation successful
0620 15:19:31.272815 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":337,"threads_started":1}
I20260620 15:19:31.277289 14987 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34597
I20260620 15:19:31.277594 14987 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:31.278251 14987 heartbeater.cc:511] Master 127.0.97.126:34597 requested a full tablet report, sending...
I20260620 15:19:31.280310 14801 ts_manager.cc:194] Registered new tserver with Master: e04b27eca0d640b68dab090557fe213c (127.0.97.65:36825)
I20260620 15:19:31.281780 14801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:57681
W20260620 15:19:31.588286 14991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:31.588557 14991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:31.588686 14991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:31.595870 14991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:31.595968 14991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:31.596154 14991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:19:31.605608 14991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:43137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=40863
--tserver_master_addrs=127.0.97.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:31.606843 14991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:31.608119 14991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:31.620285 14996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:31.620355 14997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:31.620342 14999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:31.621367 14991 server_base.cc:1034] running on GCE node
I20260620 15:19:31.621899 14991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:31.622808 14991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:31.623999 14991 hybrid_clock.cc:648] HybridClock initialized: now 1781968771623956 us; error 37 us; skew 500 ppm
I20260620 15:19:31.624259 14991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:31 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:31.627377 14991 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:19:31.628738 14991 webserver.cc:460] Webserver started at http://127.0.97.66:40863/ using document root <none> and password file <none>
I20260620 15:19:31.629318 14991 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:31.629424 14991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:31.634305 14991 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:19:31.636801 15006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:31.638180 14991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:19:31.638372 14991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "b46ebe9773b2452a9e019a8600a015c9"
format_stamp: "Formatted at 2026-06-20 15:19:28 on dist-test-slave-7s64"
server_key: "cce581704c8b02246d6efc179afe9433"
server_key_iv: "4d9141976fc7923e4c622861871e00c8"
server_key_version: "encryptionkey@0"
I20260620 15:19:31.639353 14991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:31.664252 14991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:31.668465 14991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:31.668807 14991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:31.669708 14991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:31.670930 14991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:31.671016 14991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:31.671149 14991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:31.671236 14991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:31.703168 14991 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:43137
I20260620 15:19:31.703266 15118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:43137 every 8 connection(s)
I20260620 15:19:31.704615 14991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:19:31.714020 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 14991
I20260620 15:19:31.714686 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:46641
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:34597
--webserver_port=43237
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:19:31 dist-test-slave-7s64 krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968771, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:31.725194 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:31.707092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:37013 (local address 127.0.97.126:34597)
0620 15:19:31.707267 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:31.707271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:31.708437 (+ 1166us) server_negotiation.cc:407] Connection header received
0620 15:19:31.710333 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:31.710342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:31.710423 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:31.710512 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:31.711856 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:31.712618 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:31.714427 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:31.714897 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:31.715001 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:31.719245 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:31.719250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:31.719254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:31.719286 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:31.721940 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:31.722780 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:31.722790 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:31.722792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:31.722910 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:31.723448 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:31.723453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:31.723457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:31.723785 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:31.723897 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:31.724727 (+ 830us) server_negotiation.cc:299] Negotiation successful
0620 15:19:31.724891 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:19:31.726637 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:31.707544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:37013)
0620 15:19:31.708293 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:31.708338 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:31.709988 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:31.710711 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:31.710718 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:31.711190 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:31.711705 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:31.711721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:31.712753 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:31.712757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:31.714285 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:31.714300 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:31.715115 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:31.715124 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:31.715511 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:31.716219 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:31.716253 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:31.719022 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:31.722088 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:31.722097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:31.722112 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:31.722650 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:31.723083 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:31.723091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:31.723098 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:31.723330 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:31.723904 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:31.723914 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:31.724315 (+ 401us) client_negotiation.cc:769] Sending connection context
0620 15:19:31.724664 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 15:19:31.725257 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":268,"threads_started":1}
I20260620 15:19:31.728034 15119 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34597
I20260620 15:19:31.728336 15119 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:31.729077 15119 heartbeater.cc:511] Master 127.0.97.126:34597 requested a full tablet report, sending...
I20260620 15:19:31.730434 14801 ts_manager.cc:194] Registered new tserver with Master: b46ebe9773b2452a9e019a8600a015c9 (127.0.97.66:43137)
I20260620 15:19:31.731233 14801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:37013
W20260620 15:19:32.032284 15122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:32.032598 15122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:32.032768 15122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:32.039978 15122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:32.040120 15122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:32.040300 15122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:19:32.049723 15122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:46641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=43237
--tserver_master_addrs=127.0.97.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:32.050877 15122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:32.052112 15122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:32.065021 15128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:32.065049 15130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:32.065207 15127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:32.069813 15122 server_base.cc:1034] running on GCE node
I20260620 15:19:32.070359 15122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:32.071233 15122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:32.072448 15122 hybrid_clock.cc:648] HybridClock initialized: now 1781968772072424 us; error 39 us; skew 500 ppm
I20260620 15:19:32.072811 15122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:32 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:32.075896 15122 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:19:32.077337 15122 webserver.cc:460] Webserver started at http://127.0.97.67:43237/ using document root <none> and password file <none>
I20260620 15:19:32.078009 15122 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:32.078162 15122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:32.083101 15122 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:19:32.085646 15137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:32.086884 15122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:19:32.087054 15122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "9020128448114a8fab0c511359d980cf"
format_stamp: "Formatted at 2026-06-20 15:19:29 on dist-test-slave-7s64"
server_key: "ae4346d44bf4e2326c893dc8f13040bb"
server_key_iv: "240e1e683461c1b65150409e0e34099b"
server_key_version: "encryptionkey@0"
I20260620 15:19:32.087975 15122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:32.112000 15122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:32.115548 15122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:32.115830 15122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:32.117081 15122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:32.118765 15122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:32.118911 15122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:32.119047 15122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:32.119153 15122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:32.154542 15122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:46641
I20260620 15:19:32.154623 15249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:46641 every 8 connection(s)
I20260620 15:19:32.155890 15122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:19:32.157617 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 15122
Jun 20 15:19:32 dist-test-slave-7s64 krb5kdc[14216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:32.175124 15254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.159669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.0.97.126:34597)
0620 15:19:32.160114 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.160120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.160142 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:19:32.160222 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.160230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.160323 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.160458 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:32.162417 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.163188 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.164754 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.165195 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.165348 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.169041 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:32.169045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:32.169048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:32.169078 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:32.172163 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:32.173126 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:32.173130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:32.173132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:32.173299 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:32.173676 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:32.173681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:32.173684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:32.173946 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:32.174222 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.174507 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.174719 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":367,"threads_started":1}
I20260620 15:19:32.176302 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.159480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:36863 (local address 127.0.97.126:34597)
0620 15:19:32.159983 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.160000 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.160024 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:19:32.161722 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.161729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.161812 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.161922 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:32.163280 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.164000 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.165549 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.166130 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.166256 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.170335 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:32.170340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:32.170343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:32.170383 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:32.173438 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:32.174106 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:32.174115 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:32.174117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:32.174226 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:32.174646 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:32.174655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:32.174657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:32.174995 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:32.175135 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.175797 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.175994 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353}
I20260620 15:19:32.177381 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.158891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:36863)
0620 15:19:32.159674 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.159722 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:32.161460 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:32.162117 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:32.162124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:32.162589 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:32.163095 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.163111 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.164158 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.164164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.165407 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.165419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.166394 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.166403 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.166754 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.167368 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:32.167400 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:32.170111 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:32.173569 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:32.173576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:32.173590 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:32.173981 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:32.174333 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:32.174339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:32.174342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:32.174518 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:32.175105 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:32.175113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:32.175457 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 15:19:32.175699 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 15:19:32.176340 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":341,"threads_started":1}
I20260620 15:19:32.178838 15250 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34597
I20260620 15:19:32.179080 15250 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:32.179693 15250 heartbeater.cc:511] Master 127.0.97.126:34597 requested a full tablet report, sending...
I20260620 15:19:32.180962 14801 ts_manager.cc:194] Registered new tserver with Master: 9020128448114a8fab0c511359d980cf (127.0.97.67:46641)
I20260620 15:19:32.181732 14801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:36863
I20260620 15:19:32.193970 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:19:32.200525 14989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.194868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.0.97.126:34597)
0620 15:19:32.195076 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.195083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.195402 (+ 319us) server_negotiation.cc:407] Connection header received
0620 15:19:32.195609 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.195613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.195762 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.195869 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:19:32.196279 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.197016 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.198386 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.198785 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.198871 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.199376 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:19:32.199809 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:19:32.199906 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.200047 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.200120 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:19:32.201364 14801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:19:32.208238 14801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:19:32.234159 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.219512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.0.1:34980)
0620 15:19:32.221285 (+ 1773us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.221324 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:32.221608 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:32.222914 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:32.222927 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:32.222973 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:32.223521 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.223540 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.226548 (+ 3008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.226558 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.229902 (+ 3344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.229933 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.233019 (+ 3086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.233030 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.233332 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.233407 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 15:19:32.233571 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 15:19:32.233750 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1534,"threads_started":1}
I20260620 15:19:32.234619 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.219544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.0.97.65:36825)
0620 15:19:32.220439 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.220452 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.221436 (+ 984us) server_negotiation.cc:407] Connection header received
0620 15:19:32.221886 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.221899 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.222228 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.222430 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:32.223820 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.226265 (+ 2445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.230162 (+ 3897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.232294 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.232440 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.232754 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.234077 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.234276 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":1013,"threads_started":1}
I20260620 15:19:32.239022 14922 tablet_service.cc:1467] Processing CreateTablet for tablet e4099c81fe4242f78f0886b166b1d041 (DEFAULT_TABLE table=test-table [id=03b93695453842baa539bc40970339f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:32.240590 14922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4099c81fe4242f78f0886b166b1d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:32.242797 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.220261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:46641 (local address 127.0.0.1:44942)
0620 15:19:32.234628 (+ 14367us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.234649 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:32.234955 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:32.235941 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:32.235946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:32.235973 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:32.236159 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.236174 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.238026 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.238033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.240012 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.240039 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.241838 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.241845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.242046 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.242104 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 15:19:32.242234 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 15:19:32.242371 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14236,"mutex_wait_us":31,"thread_start_us":987,"threads_started":1}
I20260620 15:19:32.242847 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.220969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.0.97.67:46641)
0620 15:19:32.233792 (+ 12823us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.233801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.234953 (+ 1152us) server_negotiation.cc:407] Connection header received
0620 15:19:32.235183 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.235192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.235415 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.235621 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:32.236397 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.237872 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.240230 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.241654 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.241817 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.242062 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.242273 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.242437 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12656,"thread_start_us":12828,"threads_started":1}
I20260620 15:19:32.244414 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.220795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.0.97.66:43137)
0620 15:19:32.235386 (+ 14591us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.235395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.235428 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:19:32.235527 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.235537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.235755 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.235946 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:32.237356 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.238794 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.241383 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.243113 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.243246 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.243426 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.243867 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.244067 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14432,"thread_start_us":14324,"threads_started":1}
I20260620 15:19:32.245230 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.219952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43137 (local address 127.0.0.1:34740)
0620 15:19:32.234543 (+ 14591us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.234564 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:32.234704 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:32.236685 (+ 1981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:32.236692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:32.236723 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:32.237107 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.237121 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.238947 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.238953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.241208 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.241225 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.243393 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.243404 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.243603 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.243640 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:19:32.243728 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 15:19:32.244836 (+ 1108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14441,"mutex_wait_us":3,"thread_start_us":16339,"threads_started":1}
I20260620 15:19:32.246824 15185 tablet_service.cc:1467] Processing CreateTablet for tablet e4099c81fe4242f78f0886b166b1d041 (DEFAULT_TABLE table=test-table [id=03b93695453842baa539bc40970339f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:32.248126 15185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4099c81fe4242f78f0886b166b1d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:32.248765 15054 tablet_service.cc:1467] Processing CreateTablet for tablet e4099c81fe4242f78f0886b166b1d041 (DEFAULT_TABLE table=test-table [id=03b93695453842baa539bc40970339f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:32.249907 15054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4099c81fe4242f78f0886b166b1d041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:32.262108 15266 tablet_bootstrap.cc:492] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: Bootstrap starting.
I20260620 15:19:32.265162 15268 tablet_bootstrap.cc:492] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: Bootstrap starting.
I20260620 15:19:32.266121 15267 tablet_bootstrap.cc:492] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: Bootstrap starting.
I20260620 15:19:32.266362 15266 tablet_bootstrap.cc:654] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:32.267633 15266 log.cc:824] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:32.268690 15268 tablet_bootstrap.cc:654] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:32.269631 15268 log.cc:824] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:32.270143 15267 tablet_bootstrap.cc:654] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:32.270711 15266 tablet_bootstrap.cc:492] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: No bootstrap required, opened a new log
I20260620 15:19:32.271024 15266 ts_tablet_manager.cc:1397] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 15:19:32.271328 15267 log.cc:824] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:32.272250 15268 tablet_bootstrap.cc:492] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: No bootstrap required, opened a new log
I20260620 15:19:32.272579 15268 ts_tablet_manager.cc:1397] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 15:19:32.277558 15267 tablet_bootstrap.cc:492] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: No bootstrap required, opened a new log
I20260620 15:19:32.277798 15267 ts_tablet_manager.cc:1397] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260620 15:19:32.279938 15268 raft_consensus.cc:348] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } }
I20260620 15:19:32.280304 15268 raft_consensus.cc:374] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:32.280400 15268 raft_consensus.cc:729] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04b27eca0d640b68dab090557fe213c, State: Initialized, Role: FOLLOWER
I20260620 15:19:32.281098 15268 consensus_queue.cc:260] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } }
I20260620 15:19:32.282167 15266 raft_consensus.cc:348] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } }
I20260620 15:19:32.282534 15266 raft_consensus.cc:374] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:32.282639 15266 raft_consensus.cc:729] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9020128448114a8fab0c511359d980cf, State: Initialized, Role: FOLLOWER
I20260620 15:19:32.283162 15266 consensus_queue.cc:260] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } }
I20260620 15:19:32.284329 14987 heartbeater.cc:503] Master 127.0.97.126:34597 was elected leader, sending a full tablet report...
I20260620 15:19:32.284315 15250 heartbeater.cc:503] Master 127.0.97.126:34597 was elected leader, sending a full tablet report...
I20260620 15:19:32.284852 15266 ts_tablet_manager.cc:1428] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 15:19:32.286562 15268 ts_tablet_manager.cc:1428] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: Time spent starting tablet: real 0.014s user 0.000s sys 0.011s
I20260620 15:19:32.287554 15267 raft_consensus.cc:348] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } }
I20260620 15:19:32.287887 15267 raft_consensus.cc:374] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:32.288004 15267 raft_consensus.cc:729] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46ebe9773b2452a9e019a8600a015c9, State: Initialized, Role: FOLLOWER
I20260620 15:19:32.288642 15267 consensus_queue.cc:260] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } }
I20260620 15:19:32.289767 15119 heartbeater.cc:503] Master 127.0.97.126:34597 was elected leader, sending a full tablet report...
I20260620 15:19:32.290297 15267 ts_tablet_manager.cc:1428] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 15:19:32.378578 15273 raft_consensus.cc:482] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:19:32.378881 15273 raft_consensus.cc:504] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } }
I20260620 15:19:32.380594 15273 leader_election.cc:288] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e04b27eca0d640b68dab090557fe213c (127.0.97.65:36825), b46ebe9773b2452a9e019a8600a015c9 (127.0.97.66:43137)
I20260620 15:19:32.386437 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.380992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:50685 (local address 127.0.97.66:43137)
0620 15:19:32.381206 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.381211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.381228 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:19:32.381431 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.381435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.381493 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.381567 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:32.382169 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.383191 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.384835 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.385588 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.385728 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.385818 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.386068 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.386193 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:19:32.386466 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.380880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43137 (local address 127.0.97.67:50685)
0620 15:19:32.381110 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.381154 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:32.381288 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:32.381637 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:32.381647 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:32.381697 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:32.381978 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.381990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.383320 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.383325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.384688 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.384700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.385750 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.385756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.385919 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.385943 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:19:32.386045 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:19:32.386171 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260620 15:19:32.387244 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.381084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:58231 (local address 127.0.97.65:36825)
0620 15:19:32.381318 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.381335 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.382172 (+ 837us) server_negotiation.cc:407] Connection header received
0620 15:19:32.382358 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.382362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.382422 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.382526 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:32.383083 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.383771 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.385534 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.386304 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.386464 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.386555 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.386878 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.387002 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:19:32.387343 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.380880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.97.67:58231)
0620 15:19:32.382082 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.382100 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:32.382225 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:32.382535 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:32.382540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:32.382563 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:32.382903 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.382913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.383942 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.383948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.385376 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:32.385388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.386515 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.386521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:32.386700 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.386743 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 15:19:32.386850 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 15:19:32.386964 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":547,"thread_start_us":344,"threads_started":1}
I20260620 15:19:32.387360 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4099c81fe4242f78f0886b166b1d041" candidate_uuid: "9020128448114a8fab0c511359d980cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46ebe9773b2452a9e019a8600a015c9" is_pre_election: true
I20260620 15:19:32.387868 15074 raft_consensus.cc:2454] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9020128448114a8fab0c511359d980cf in term 0.
I20260620 15:19:32.388382 14942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4099c81fe4242f78f0886b166b1d041" candidate_uuid: "9020128448114a8fab0c511359d980cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04b27eca0d640b68dab090557fe213c" is_pre_election: true
I20260620 15:19:32.388605 15141 leader_election.cc:302] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9020128448114a8fab0c511359d980cf, b46ebe9773b2452a9e019a8600a015c9; no voters:
I20260620 15:19:32.388912 14942 raft_consensus.cc:2454] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9020128448114a8fab0c511359d980cf in term 0.
I20260620 15:19:32.389046 15273 raft_consensus.cc:2781] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:19:32.389176 15273 raft_consensus.cc:482] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:19:32.389287 15273 raft_consensus.cc:3037] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:32.391409 15273 raft_consensus.cc:504] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } }
I20260620 15:19:32.392009 15273 leader_election.cc:288] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [CANDIDATE]: Term 1 election: Requested vote from peers e04b27eca0d640b68dab090557fe213c (127.0.97.65:36825), b46ebe9773b2452a9e019a8600a015c9 (127.0.97.66:43137)
I20260620 15:19:32.392403 14942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4099c81fe4242f78f0886b166b1d041" candidate_uuid: "9020128448114a8fab0c511359d980cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e04b27eca0d640b68dab090557fe213c"
I20260620 15:19:32.392560 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4099c81fe4242f78f0886b166b1d041" candidate_uuid: "9020128448114a8fab0c511359d980cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46ebe9773b2452a9e019a8600a015c9"
I20260620 15:19:32.392622 14942 raft_consensus.cc:3037] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:32.392763 15074 raft_consensus.cc:3037] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:32.394831 14942 raft_consensus.cc:2454] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9020128448114a8fab0c511359d980cf in term 1.
I20260620 15:19:32.395009 15074 raft_consensus.cc:2454] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9020128448114a8fab0c511359d980cf in term 1.
I20260620 15:19:32.395151 15138 leader_election.cc:302] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9020128448114a8fab0c511359d980cf, e04b27eca0d640b68dab090557fe213c; no voters:
I20260620 15:19:32.395450 15273 raft_consensus.cc:2781] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:19:32.395870 15273 raft_consensus.cc:686] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 1 LEADER]: Becoming Leader. State: Replica: 9020128448114a8fab0c511359d980cf, State: Running, Role: LEADER
I20260620 15:19:32.396445 15273 consensus_queue.cc:237] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } }
I20260620 15:19:32.401029 14801 catalog_manager.cc:5495] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9020128448114a8fab0c511359d980cf (127.0.97.67). New cstate: current_term: 1 leader_uuid: "9020128448114a8fab0c511359d980cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9020128448114a8fab0c511359d980cf" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46641 } health_report { overall_health: HEALTHY } } }
W20260620 15:19:32.407418 15251 tablet.cc:2369] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:19:32.441057 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.436049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.0.97.67:46641)
0620 15:19:32.436226 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.436230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.436247 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:19:32.436333 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.436336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.436408 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.436581 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:19:32.436888 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.437570 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.438962 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.439335 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.439422 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.439933 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:19:32.440404 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:19:32.440524 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.440729 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.440790 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:19:32.448664 14942 raft_consensus.cc:1261] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c [term 1 FOLLOWER]: Refusing update from remote peer 9020128448114a8fab0c511359d980cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:19:32.448765 15074 raft_consensus.cc:1261] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9 [term 1 FOLLOWER]: Refusing update from remote peer 9020128448114a8fab0c511359d980cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:19:32.449635 15276 consensus_queue.cc:1035] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e04b27eca0d640b68dab090557fe213c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:19:32.449955 15273 consensus_queue.cc:1035] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b46ebe9773b2452a9e019a8600a015c9" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:19:32.456676 15120 tablet.cc:2369] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:19:32.461263 15279 mvcc.cc:205] Tried to move back new op lower bound from 7298944091940581376 to 7298944091739807744. Current Snapshot: MvccSnapshot[applied={T|T < 7298944091940581376}]
I20260620 15:19:32.473259 14801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:19:32.473480 14801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:19:32.476745 14801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Sending DeleteTablet for 3 replicas of tablet e4099c81fe4242f78f0886b166b1d041
I20260620 15:19:32.477794 15054 tablet_service.cc:1514] Processing DeleteTablet for tablet e4099c81fe4242f78f0886b166b1d041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:32 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34740
I20260620 15:19:32.477854 14922 tablet_service.cc:1514] Processing DeleteTablet for tablet e4099c81fe4242f78f0886b166b1d041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:32 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34980
I20260620 15:19:32.478011 15185 tablet_service.cc:1514] Processing DeleteTablet for tablet e4099c81fe4242f78f0886b166b1d041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:32 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44942
I20260620 15:19:32.478667 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 14838
I20260620 15:19:32.484952 15289 tablet_replica.cc:330] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: stopping tablet replica
I20260620 15:19:32.485364 15288 tablet_replica.cc:330] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: stopping tablet replica
I20260620 15:19:32.485400 15289 raft_consensus.cc:2229] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:19:32.485829 15289 raft_consensus.cc:2258] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:32.485951 15290 tablet_replica.cc:330] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: stopping tablet replica
I20260620 15:19:32.486335 15290 raft_consensus.cc:2229] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:19:32.486835 15288 raft_consensus.cc:2229] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:19:32.487241 15288 raft_consensus.cc:2258] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:32.488020 15290 raft_consensus.cc:2258] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:32.489920 15290 ts_tablet_manager.cc:1905] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:32.491762 15289 ts_tablet_manager.cc:1905] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:32.492026 15288 ts_tablet_manager.cc:1905] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:32.497505 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.479603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.0.97.65:36825)
0620 15:19:32.479827 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.479836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.479863 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:19:32.479963 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.479972 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.480071 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.480215 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:32.481991 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.483625 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.485548 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.486320 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.486477 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.489891 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:32.489922 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:32.489938 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:32.489998 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:32.494111 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:32.494949 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:32.494958 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:32.494962 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:32.495133 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:32.495576 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:32.495581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:32.495584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:32.496111 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:32.496414 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.496840 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.497081 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:19:32.498438 14982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:32.503374 15288 ts_tablet_manager.cc:1918] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:32.503590 15288 log.cc:1196] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/e4099c81fe4242f78f0886b166b1d041
I20260620 15:19:32.504855 15288 ts_tablet_manager.cc:1939] T e4099c81fe4242f78f0886b166b1d041 P b46ebe9773b2452a9e019a8600a015c9: Deleting consensus metadata
I20260620 15:19:32.506675 14788 catalog_manager.cc:4841] TS b46ebe9773b2452a9e019a8600a015c9 (127.0.97.66:43137): tablet e4099c81fe4242f78f0886b166b1d041 (table test-table [id=03b93695453842baa539bc40970339f3]) successfully deleted
I20260620 15:19:32.510236 15290 ts_tablet_manager.cc:1918] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:32.510426 15290 log.cc:1196] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/e4099c81fe4242f78f0886b166b1d041
I20260620 15:19:32.510886 15290 ts_tablet_manager.cc:1939] T e4099c81fe4242f78f0886b166b1d041 P 9020128448114a8fab0c511359d980cf: Deleting consensus metadata
I20260620 15:19:32.512259 14786 catalog_manager.cc:4841] TS 9020128448114a8fab0c511359d980cf (127.0.97.67:46641): tablet e4099c81fe4242f78f0886b166b1d041 (table test-table [id=03b93695453842baa539bc40970339f3]) successfully deleted
I20260620 15:19:32.614383 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 14838
I20260620 15:19:32.638077 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 14991
W20260620 15:19:32.638136 14785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:36825 (error 108)
I20260620 15:19:32.640832 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.640235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.0.1:35002)
0620 15:19:32.640525 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.640646 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260620 15:19:32.641104 14785 catalog_manager.cc:4584] TS e04b27eca0d640b68dab090557fe213c (127.0.97.65:36825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4099c81fe4242f78f0886b166b1d041: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36825: connect: Connection refused (error 111)
I20260620 15:19:32.654682 15264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.639022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.0.97.66:43137)
0620 15:19:32.639237 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.639246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.639545 (+ 299us) server_negotiation.cc:407] Connection header received
0620 15:19:32.639662 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.639667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.639752 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.639864 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:32.641655 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.642875 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.644328 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.645149 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.645307 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.647850 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:32.647871 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:32.647888 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:32.647947 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:32.651664 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:32.652319 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:32.652329 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:32.652333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:32.652621 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:32.653047 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:32.653053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:32.653056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:32.653549 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:32.653846 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.654040 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.654270 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:19:32.655359 15114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:32.694796 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.694270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.0.1:35014)
0620 15:19:32.694505 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.694608 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 15:19:32.695961 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.695537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.0.1:35030)
0620 15:19:32.695713 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.695814 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 15:19:32.745680 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 14991
I20260620 15:19:32.756736 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.756201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.0.1:35032)
0620 15:19:32.756443 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.756543 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 15:19:32.758008 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.757636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.0.1:35038)
0620 15:19:32.757814 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.757886 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 15:19:32.768848 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 15122
I20260620 15:19:32.784700 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.769614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.0.97.67:46641)
0620 15:19:32.769813 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:32.769819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:32.769840 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:19:32.770021 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:32.770026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:32.770122 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:32.770251 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:32.771863 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.773181 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.774563 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:32.775281 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:32.775422 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:32.778029 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:32.778042 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:32.778060 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:32.778115 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:32.781872 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:32.782583 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:32.782594 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:32.782603 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:32.782785 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:32.783112 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:32.783118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:32.783121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:32.783556 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:32.783850 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:32.784012 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 15:19:32.784236 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:19:32.785583 15245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:32.838716 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.838171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.0.1:35046)
0620 15:19:32.838409 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.838519 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 15:19:32.839866 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:32.839433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.0.1:35054)
0620 15:19:32.839651 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:32.839742 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 15:19:32.909300 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 15122
I20260620 15:19:32.928848 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 14768
I20260620 15:19:32.929971 14831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:33.040436 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:33.039645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36825 (local address 127.0.0.1:35064)
0620 15:19:33.040098 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:33.040201 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266}
I20260620 15:19:33.040488 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 14768
I20260620 15:19:33.062330 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34597
--webserver_interface=127.0.97.126
--webserver_port=39755
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:34597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:33.380978 15296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:33.381232 15296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:33.381377 15296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:33.388367 15296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:19:33.388461 15296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:33.388566 15296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:33.388664 15296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:19:33.388731 15296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:19:33.398168 15296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:34597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:33.399344 15296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:33.400651 15296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:33.411899 15302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:33.411959 15304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:33.411923 15301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:33.413070 15296 server_base.cc:1034] running on GCE node
I20260620 15:19:33.413832 15296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:33.415189 15296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:33.416412 15296 hybrid_clock.cc:648] HybridClock initialized: now 1781968773416361 us; error 57 us; skew 500 ppm
I20260620 15:19:33.416750 15296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:33 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:33.419628 15296 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:19:33.421106 15296 webserver.cc:460] Webserver started at http://127.0.97.126:39755/ using document root <none> and password file <none>
I20260620 15:19:33.421816 15296 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:33.421957 15296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:33.427191 15296 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:19:33.430052 15311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:33.431344 15296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 15:19:33.431566 15296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f"
format_stamp: "Formatted at 2026-06-20 15:19:27 on dist-test-slave-7s64"
server_key: "d959f9bef1d0b05564ba901b9a46acf7"
server_key_iv: "8d2807944e243662c34afc93ad1fe3dd"
server_key_version: "encryptionkey@0"
I20260620 15:19:33.432569 15296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:33.457588 15296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:33.461633 15296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:33.461901 15296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:33.480026 15296 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:34597
I20260620 15:19:33.480121 15362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:34597 every 8 connection(s)
I20260620 15:19:33.481480 15296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:19:33.482851 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 15296
I20260620 15:19:33.483855 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:36825
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:34597
--webserver_port=46515
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:19:33.491012 15363 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:19:33.494401 15363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Bootstrap starting.
I20260620 15:19:33.514111 15363 log.cc:824] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:33.529261 15363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:19:33.529760 15363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f: Bootstrap complete.
I20260620 15:19:33.538818 15363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:33.539659 15363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8e5cd4a60b4d26bd2f81b0156aab0f, State: Initialized, Role: FOLLOWER
I20260620 15:19:33.540320 15363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:33.540558 15363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:19:33.540675 15363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:19:33.540853 15363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 2 FOLLOWER]: Advancing to term 3
I20260620 15:19:33.544358 15363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:33.544823 15363 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8e5cd4a60b4d26bd2f81b0156aab0f; no voters:
I20260620 15:19:33.545401 15363 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 15:19:33.545598 15367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 3 FOLLOWER]: Leader election won for term 3
I20260620 15:19:33.546656 15367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [term 3 LEADER]: Becoming Leader. State: Replica: 0d8e5cd4a60b4d26bd2f81b0156aab0f, State: Running, Role: LEADER
I20260620 15:19:33.547233 15363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:19:33.547261 15367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } }
I20260620 15:19:33.550060 15369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8e5cd4a60b4d26bd2f81b0156aab0f. Latest consensus state: current_term: 3 leader_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } } }
I20260620 15:19:33.550468 15369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:33.551075 15368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8e5cd4a60b4d26bd2f81b0156aab0f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34597 } } }
I20260620 15:19:33.551405 15368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8e5cd4a60b4d26bd2f81b0156aab0f [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:33.561369 15375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:19:33.571327 15375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:19:33.573478 15375 catalog_manager.cc:1259] Loaded cluster ID: cd5e0c71ed1b450899120332f6649a33
I20260620 15:19:33.573661 15375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 15:19:33.576865 15375 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa0543f99ad google::LogMessage::Fail() at ??:0
I20260620 15:19:33.580726 15384 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fa0543fc88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa0543f94de google::LogMessage::Flush() at ??:0
@ 0x7fa0543faa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa069e2c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa069e29976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa069eb1610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa069eb1439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa057045345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa05704891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa057048799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa057023eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa05a2fd6db start_thread at ??:0
@ 0x7fa05329361f clone at ??:0
W20260620 15:19:33.883824 15365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:33.884076 15365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:33.884167 15365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:33.891135 15365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:33.891225 15365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:33.891331 15365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:19:33.900615 15365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:36825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=46515
--tserver_master_addrs=127.0.97.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:33.901748 15365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:33.902974 15365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:33.915746 15404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:33.915779 15407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:33.915817 15405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:33.916886 15365 server_base.cc:1034] running on GCE node
I20260620 15:19:33.917455 15365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:33.918331 15365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:33.919546 15365 hybrid_clock.cc:648] HybridClock initialized: now 1781968773919506 us; error 64 us; skew 500 ppm
I20260620 15:19:33.919883 15365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:33 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:33.922956 15365 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:19:33.924314 15365 webserver.cc:460] Webserver started at http://127.0.97.65:46515/ using document root <none> and password file <none>
I20260620 15:19:33.925029 15365 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:33.925176 15365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:33.930290 15365 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:19:33.933161 15414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:33.934378 15365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:19:33.934551 15365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "e04b27eca0d640b68dab090557fe213c"
format_stamp: "Formatted at 2026-06-20 15:19:28 on dist-test-slave-7s64"
server_key: "085af855fd0917f10961b5a48bc1d4a0"
server_key_iv: "553cc50d378006fa0577b11bfe8e1865"
server_key_version: "encryptionkey@0"
I20260620 15:19:33.935477 15365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:33.950948 15365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:33.954550 15365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:33.954806 15365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:33.955683 15365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:33.957437 15421 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 15:19:33.962399 15421 ts_tablet_manager.cc:1852] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 15:19:33.963276 15421 ts_tablet_manager.cc:1905] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:33.969964 15421 ts_tablet_manager.cc:1918] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:33.970110 15421 log.cc:1196] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/e4099c81fe4242f78f0886b166b1d041
I20260620 15:19:33.970480 15421 ts_tablet_manager.cc:1939] T e4099c81fe4242f78f0886b166b1d041 P e04b27eca0d640b68dab090557fe213c: Deleting consensus metadata
I20260620 15:19:33.970788 15365 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 15:19:33.970916 15365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.001s
I20260620 15:19:33.971058 15365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:33.971143 15365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:34.006198 15365 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36825
I20260620 15:19:34.006283 15527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36825 every 8 connection(s)
I20260620 15:19:34.007541 15365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:19:34.008836 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 15365
I20260620 15:19:34.009593 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:43137
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:34597
--webserver_port=40863
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:19:34.012940 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:34.011019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:43741)
0620 15:19:34.011823 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:34.012057 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":330,"threads_started":1}
W20260620 15:19:34.013732 15528 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:34597 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:34597: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
I20260620 15:19:34.014640 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:34.014256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:47595)
0620 15:19:34.014422 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:34.014492 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:19:34.015542 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:34.015221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:49103)
0620 15:19:34.015355 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:34.015432 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:19:34.015746 15528 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:19:34.325548 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:34.325793 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:34.325881 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:34.333307 15531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:34.333397 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:34.333501 15531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:19:34.342799 15531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:43137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=40863
--tserver_master_addrs=127.0.97.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:34.343842 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:34.345069 15531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:34.357583 15536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:34.357591 15537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:34.357493 15539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:34.358464 15531 server_base.cc:1034] running on GCE node
I20260620 15:19:34.359016 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:34.359942 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:34.361158 15531 hybrid_clock.cc:648] HybridClock initialized: now 1781968774361119 us; error 71 us; skew 500 ppm
I20260620 15:19:34.361451 15531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:34 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:34.364358 15531 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:19:34.365748 15531 webserver.cc:460] Webserver started at http://127.0.97.66:40863/ using document root <none> and password file <none>
I20260620 15:19:34.366335 15531 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:34.366431 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:34.371464 15531 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260620 15:19:34.374069 15546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:34.375386 15531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 15:19:34.375531 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "b46ebe9773b2452a9e019a8600a015c9"
format_stamp: "Formatted at 2026-06-20 15:19:28 on dist-test-slave-7s64"
server_key: "cce581704c8b02246d6efc179afe9433"
server_key_iv: "4d9141976fc7923e4c622861871e00c8"
server_key_version: "encryptionkey@0"
I20260620 15:19:34.376456 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:34.394198 15531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:34.397831 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:34.398038 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:34.398855 15531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:34.400046 15531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:34.400185 15531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:34.400319 15531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:34.400400 15531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:34.433147 15531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:43137
I20260620 15:19:34.433226 15658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:43137 every 8 connection(s)
I20260620 15:19:34.434504 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:19:34.439103 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:34.437310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:53527)
0620 15:19:34.438018 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:34.438213 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":293,"threads_started":1}
W20260620 15:19:34.439683 15659 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:34597 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:34597: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
I20260620 15:19:34.440389 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:34.440095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:52043)
0620 15:19:34.440218 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:34.440289 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 15:19:34.441286 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:34.440860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:55841)
0620 15:19:34.441047 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:34.441144 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 15:19:34.441411 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 15531
W20260620 15:19:34.441520 15659 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:19:34.442039 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:46641
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:34597
--webserver_port=43237
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:34.753223 15662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:34.753463 15662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:34.753559 15662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:34.760418 15662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:34.760548 15662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:34.760684 15662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:19:34.769737 15662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:46641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=43237
--tserver_master_addrs=127.0.97.126:34597
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:34.770834 15662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:34.771997 15662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:34.784336 15670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:34.784341 15668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:34.784579 15667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:34.785326 15662 server_base.cc:1034] running on GCE node
I20260620 15:19:34.785874 15662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:34.786722 15662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:34.787981 15662 hybrid_clock.cc:648] HybridClock initialized: now 1781968774787933 us; error 52 us; skew 500 ppm
I20260620 15:19:34.788314 15662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:34 dist-test-slave-7s64 krb5kdc[14216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:34.791471 15662 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:19:34.792976 15662 webserver.cc:460] Webserver started at http://127.0.97.67:43237/ using document root <none> and password file <none>
I20260620 15:19:34.793637 15662 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:34.793780 15662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:34.798923 15662 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 15:19:34.801716 15677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:34.802958 15662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:19:34.803128 15662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "9020128448114a8fab0c511359d980cf"
format_stamp: "Formatted at 2026-06-20 15:19:29 on dist-test-slave-7s64"
server_key: "ae4346d44bf4e2326c893dc8f13040bb"
server_key_iv: "240e1e683461c1b65150409e0e34099b"
server_key_version: "encryptionkey@0"
I20260620 15:19:34.804081 15662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:34.815325 15662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:34.819373 15662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:34.819614 15662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:34.820562 15662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:34.821854 15662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:34.821965 15662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:34.822085 15662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:34.822191 15662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:34.854773 15662 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:46641
I20260620 15:19:34.854856 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:46641 every 8 connection(s)
I20260620 15:19:34.856128 15662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:19:34.860720 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:34.858836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:47857)
0620 15:19:34.859548 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:34.859756 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":283,"threads_started":1}
W20260620 15:19:34.861413 15790 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:34597 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:34597: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
I20260620 15:19:34.862040 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 15662
I20260620 15:19:34.862159 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:34.861819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:59065)
0620 15:19:34.861962 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:34.862030 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:19:34.863063 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:34.862687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:39061)
0620 15:19:34.862841 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:34.862928 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 15:19:34.863339 15790 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:19:35.017066 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:35.016272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:46737)
0620 15:19:35.016742 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:35.016821 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":229,"threads_started":1}
I20260620 15:19:35.442785 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:35.442040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:57543)
0620 15:19:35.442487 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:35.442584 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260620 15:19:35.864529 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:35.863784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:56829)
0620 15:19:35.864203 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:35.864272 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":202,"threads_started":1}
I20260620 15:19:36.018639 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:36.017990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:45243)
0620 15:19:36.018406 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:36.018488 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":229,"threads_started":1}
I20260620 15:19:36.444245 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:36.443559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:42767)
0620 15:19:36.443997 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:36.444083 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":218,"threads_started":1}
I20260620 15:19:36.865932 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:36.865260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:36063)
0620 15:19:36.865718 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:36.865789 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
I20260620 15:19:37.020155 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:37.019494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:39773)
0620 15:19:37.019908 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:37.019987 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":210,"threads_started":1}
I20260620 15:19:37.445935 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:37.445107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:35589)
0620 15:19:37.445692 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:37.445777 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":381,"threads_started":1}
I20260620 15:19:37.867486 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:37.866861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:46031)
0620 15:19:37.867261 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:37.867332 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":216,"threads_started":1}
I20260620 15:19:38.021569 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:38.020954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:54391)
0620 15:19:38.021349 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:38.021421 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
I20260620 15:19:38.447463 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:38.446733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:39805)
0620 15:19:38.447205 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:38.447280 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":251,"threads_started":1}
I20260620 15:19:38.869035 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:38.868316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:57987)
0620 15:19:38.868776 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:38.868848 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
I20260620 15:19:39.023070 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:39.022396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:36493)
0620 15:19:39.022836 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:39.022911 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260620 15:19:39.449195 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:39.448363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:56515)
0620 15:19:39.448951 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:39.449024 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":388,"threads_started":1}
I20260620 15:19:39.870530 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:39.869883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:37867)
0620 15:19:39.870301 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:39.870374 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":218,"threads_started":1}
I20260620 15:19:40.024961 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:40.023997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:37853)
0620 15:19:40.024589 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:40.024675 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":301,"threads_started":1}
I20260620 15:19:40.450767 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:40.450067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:33687)
0620 15:19:40.450512 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:40.450595 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":241,"threads_started":1}
I20260620 15:19:40.872092 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:40.871353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:34157)
0620 15:19:40.871844 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:40.871927 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":274,"threads_started":1}
I20260620 15:19:41.026468 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:41.025748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:39307)
0620 15:19:41.026198 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:41.026271 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":191,"threads_started":1}
I20260620 15:19:41.452301 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:41.451532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:37825)
0620 15:19:41.452010 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:41.452099 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":230,"threads_started":1}
I20260620 15:19:41.873925 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:41.872989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:58615)
0620 15:19:41.873523 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:41.873652 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":234,"threads_started":1}
I20260620 15:19:42.027976 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:42.027291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:53183)
0620 15:19:42.027740 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:42.027819 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
I20260620 15:19:42.453855 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:42.453186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:34351)
0620 15:19:42.453600 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:42.453696 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":208,"threads_started":1}
I20260620 15:19:42.875517 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:42.874748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:35915)
0620 15:19:42.875293 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:42.875368 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":308,"threads_started":1}
I20260620 15:19:43.029584 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:43.028868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:57959)
0620 15:19:43.029322 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:43.029420 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":222,"threads_started":1}
I20260620 15:19:43.455348 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:43.454656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:42931)
0620 15:19:43.455108 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:43.455187 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":220,"threads_started":1}
I20260620 15:19:43.877125 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:43.876355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:37789)
0620 15:19:43.876846 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:43.876931 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":260,"threads_started":1}
I20260620 15:19:44.031185 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:44.030481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:55761)
0620 15:19:44.030926 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:44.031008 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":247,"threads_started":1}
I20260620 15:19:44.457013 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:44.456247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:54765)
0620 15:19:44.456772 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:44.456856 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":320,"threads_started":1}
I20260620 15:19:44.878697 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:44.878012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:33783)
0620 15:19:44.878444 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:44.878518 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":244,"threads_started":1}
I20260620 15:19:45.032855 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:45.032025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:49311)
0620 15:19:45.032594 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:45.032675 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":276,"threads_started":1}
I20260620 15:19:45.458580 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:45.457880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:37889)
0620 15:19:45.458321 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:45.458405 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
I20260620 15:19:45.880388 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:45.879583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:33357)
0620 15:19:45.880105 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:45.880200 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":204,"threads_started":1}
I20260620 15:19:46.034482 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:46.033716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:43633)
0620 15:19:46.034245 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:46.034322 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":293,"threads_started":1}
I20260620 15:19:46.460203 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:46.459516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:60783)
0620 15:19:46.459949 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:46.460031 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
I20260620 15:19:46.881876 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:46.881197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:50785)
0620 15:19:46.881606 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:46.881705 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":218,"threads_started":1}
I20260620 15:19:47.036090 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:47.035367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:60957)
0620 15:19:47.035819 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:47.035912 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":226,"threads_started":1}
I20260620 15:19:47.462146 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:47.461311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:39767)
0620 15:19:47.461834 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:47.461933 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":265,"threads_started":1}
I20260620 15:19:47.883401 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:47.882715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:50771)
0620 15:19:47.883127 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:47.883243 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":197,"threads_started":1}
I20260620 15:19:48.037820 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:48.037125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:45657)
0620 15:19:48.037555 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:48.037633 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":224,"threads_started":1}
I20260620 15:19:48.463822 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:48.462997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:49221)
0620 15:19:48.463472 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:48.463608 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260620 15:19:48.885237 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:48.884313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:49039)
0620 15:19:48.884852 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:48.884970 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":228,"threads_started":1}
I20260620 15:19:49.039577 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:49.038873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:46435)
0620 15:19:49.039311 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:49.039411 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":184,"threads_started":1}
I20260620 15:19:49.465677 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:49.464921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:36855)
0620 15:19:49.465370 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:49.465472 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":231,"threads_started":1}
I20260620 15:19:49.874193 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 15365
I20260620 15:19:49.887074 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:49.886381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.67:42861)
0620 15:19:49.886803 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:49.886905 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":213,"threads_started":1}
I20260620 15:19:50.041249 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:50.040456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.65:58907)
0620 15:19:50.041001 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:50.041082 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":248,"threads_started":1}
I20260620 15:19:50.467459 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:50.466694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34597 (local address 127.0.97.66:35119)
0620 15:19:50.467221 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:50.467311 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:34597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":236,"threads_started":1}
I20260620 15:19:50.597484 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:49.875418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.0.97.65:36825)
0620 15:19:49.876101 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:49.876120 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:50.581297 (+705177us) server_negotiation.cc:407] Connection header received
0620 15:19:50.583030 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:50.583040 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:50.583487 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:50.583816 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:50.585374 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:50.586587 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:50.587812 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:50.588476 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:50.588750 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:50.591338 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:50.591346 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:50.591360 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:50.591396 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:50.594372 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:50.595066 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:50.595073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:50.595079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:50.595257 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:50.595585 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:50.595588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:50.595607 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:50.596103 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:50.596338 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:50.596748 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 15:19:50.597169 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":311,"threads_started":1}
I20260620 15:19:50.598834 15523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:50.682953 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 15365
I20260620 15:19:50.700011 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 15531
I20260620 15:19:50.717904 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:50.701153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.0.97.66:43137)
0620 15:19:50.701740 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:50.701754 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:50.701794 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:19:50.703374 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:50.703383 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:50.703797 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:50.704075 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:50.705530 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:50.706855 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:50.708153 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:50.708991 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:50.709125 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:50.712002 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:50.712009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:50.712023 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:50.712061 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:50.714886 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:50.715651 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:50.715657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:50.715663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:50.715820 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:50.716176 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:50.716179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:50.716181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:50.716723 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:50.716942 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:50.717312 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 15:19:50.717599 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":261,"threads_started":1}
I20260620 15:19:50.719314 15654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:50.802970 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 15531
I20260620 15:19:50.821132 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 15662
I20260620 15:19:50.837340 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:50.822040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.0.97.67:46641)
0620 15:19:50.822580 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:50.822593 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:50.822631 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:19:50.824036 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:50.824045 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:50.824454 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:50.824840 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:50.826065 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:50.827277 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:50.828544 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:50.829194 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:50.829328 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:50.831800 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:50.831807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:50.831822 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:50.831856 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:50.834702 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:50.835243 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:50.835249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:50.835256 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:50.835399 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:50.835714 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:50.835717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:50.835719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:50.836161 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:50.836364 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:50.836749 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 15:19:50.837056 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260620 15:19:50.838606 15785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:50.921762 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 15662
2026-06-20T15:19:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24900 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:19:51 dist-test-slave-7s64 krb5kdc[15850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:19:51 dist-test-slave-7s64 krb5kdc[15850](info): set up 2 sockets
Jun 20 15:19:51 dist-test-slave-7s64 krb5kdc[15850](info): commencing operation
krb5kdc: starting...
W20260620 15:19:52.021100 389 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:19:52 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:19:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:19:52.237749 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35917
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:52.551280 15863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:52.551600 15863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:52.551766 15863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:52.558717 15863 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:19:52.558841 15863 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:19:52.558931 15863 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:19:52.559041 15863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:19:52.559135 15863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:52.559230 15863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:52.559302 15863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:19:52.559376 15863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:19:52.569041 15863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:52.570245 15863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:52.571569 15863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:52.583374 15868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:52.583350 15869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:52.583529 15871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:52.583737 15863 server_base.cc:1034] running on GCE node
I20260620 15:19:52.584818 15863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:52.586127 15863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:52.587383 15863 hybrid_clock.cc:648] HybridClock initialized: now 1781968792587323 us; error 37 us; skew 500 ppm
I20260620 15:19:52.587707 15863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:52 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:52.590905 15863 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:19:52.592398 15863 webserver.cc:460] Webserver started at http://127.0.97.126:41943/ using document root <none> and password file <none>
I20260620 15:19:52.593122 15863 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:52.593267 15863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:52.593578 15863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:52.596431 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "2ca71167a4574b0591d4a58e89ce4ba0"
format_stamp: "Formatted at 2026-06-20 15:19:52 on dist-test-slave-7s64"
server_key: "7f31729cc8c6f72f47d92d9237b6cbeb"
server_key_iv: "ddc442ee3361b1818b4ed91a45d3f18d"
server_key_version: "encryptionkey@0"
I20260620 15:19:52.597281 15863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "2ca71167a4574b0591d4a58e89ce4ba0"
format_stamp: "Formatted at 2026-06-20 15:19:52 on dist-test-slave-7s64"
server_key: "7f31729cc8c6f72f47d92d9237b6cbeb"
server_key_iv: "ddc442ee3361b1818b4ed91a45d3f18d"
server_key_version: "encryptionkey@0"
I20260620 15:19:52.602330 15863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:19:52.605825 15878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:52.607127 15863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:19:52.607331 15863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "2ca71167a4574b0591d4a58e89ce4ba0"
format_stamp: "Formatted at 2026-06-20 15:19:52 on dist-test-slave-7s64"
server_key: "7f31729cc8c6f72f47d92d9237b6cbeb"
server_key_iv: "ddc442ee3361b1818b4ed91a45d3f18d"
server_key_version: "encryptionkey@0"
I20260620 15:19:52.607546 15863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:52.626735 15863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:52.630592 15863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:52.630848 15863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:52.649374 15863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35917
I20260620 15:19:52.649410 15929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35917 every 8 connection(s)
I20260620 15:19:52.650625 15863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:19:52.654273 15930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:52.656050 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 15863
I20260620 15:19:52.656443 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:19:52.657111 389 external_mini_cluster.cc:1428] Setting key 551b58b6e2ecdd056df307b81d9ce1c1
I20260620 15:19:52.666787 15930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Bootstrap starting.
I20260620 15:19:52.670761 15930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:52.672133 15930 log.cc:824] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:19:52 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:52.675339 15930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: No bootstrap required, opened a new log
I20260620 15:19:52.681288 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:52.658964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.0.97.126:35917)
0620 15:19:52.659770 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:52.659781 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:52.659826 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:19:52.661584 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:52.661606 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:52.661964 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:52.662256 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:52.664257 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:52.665597 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:52.667636 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:52.668304 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:52.668437 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:52.673970 (+ 5533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:52.673977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:52.673991 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:52.674026 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:52.677289 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:52.678007 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:52.678013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:52.678019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:52.678177 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:52.678562 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:52.678568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:52.678571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:52.679085 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:52.679394 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:52.679871 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 15:19:52.680243 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":371,"threads_started":1}
I20260620 15:19:52.683806 15930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:19:52.684260 15930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:52.684376 15930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca71167a4574b0591d4a58e89ce4ba0, State: Initialized, Role: FOLLOWER
I20260620 15:19:52.684953 15930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:19:52.685232 15930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:19:52.685385 15930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:19:52.685578 15930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:52.688139 15930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:19:52.688772 15930 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca71167a4574b0591d4a58e89ce4ba0; no voters:
I20260620 15:19:52.689303 15930 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:19:52.689451 15935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:19:52.690142 15935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 1 LEADER]: Becoming Leader. State: Replica: 2ca71167a4574b0591d4a58e89ce4ba0, State: Running, Role: LEADER
I20260620 15:19:52.690645 15935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:19:52.691457 15930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:19:52.693418 15936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } } }
I20260620 15:19:52.693814 15936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:52.693537 15937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca71167a4574b0591d4a58e89ce4ba0. Latest consensus state: current_term: 1 leader_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } } }
I20260620 15:19:52.693979 15937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: This master's current role is: LEADER
I20260620 15:19:52.696712 15941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:19:52.702508 15941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:19:52.710975 15941 catalog_manager.cc:1347] Generated new cluster ID: 8466b4886aee46158e337fb5c47e956a
I20260620 15:19:52.711081 15941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:19:52.729305 15941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:19:52.730607 15941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:19:52.735934 15941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Generated new TSK 0
I20260620 15:19:52.736726 15941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:19:52.792229 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35917
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:53.111548 15956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:53.111822 15956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:53.111999 15956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:53.119035 15956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:53.119123 15956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:53.119334 15956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:19:53.128702 15956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:53.129873 15956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:53.131103 15956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:53.146914 15962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:53.147148 15964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:53.147369 15961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:53.147750 15956 server_base.cc:1034] running on GCE node
I20260620 15:19:53.148360 15956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:53.149323 15956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:53.150552 15956 hybrid_clock.cc:648] HybridClock initialized: now 1781968793150515 us; error 52 us; skew 500 ppm
I20260620 15:19:53.150866 15956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:53 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:53.153962 15956 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:19:53.155319 15956 webserver.cc:460] Webserver started at http://127.0.97.65:40089/ using document root <none> and password file <none>
I20260620 15:19:53.155923 15956 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:53.156016 15956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:53.156263 15956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:53.159101 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "a59e45b9301d4b21b1d72f1f36e4a247"
format_stamp: "Formatted at 2026-06-20 15:19:53 on dist-test-slave-7s64"
server_key: "e178fdaf7efc66a2f7a7c5c9d4255cb2"
server_key_iv: "c3f94ddb942a5f39253b776e69e1e6e3"
server_key_version: "encryptionkey@0"
I20260620 15:19:53.159807 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "a59e45b9301d4b21b1d72f1f36e4a247"
format_stamp: "Formatted at 2026-06-20 15:19:53 on dist-test-slave-7s64"
server_key: "e178fdaf7efc66a2f7a7c5c9d4255cb2"
server_key_iv: "c3f94ddb942a5f39253b776e69e1e6e3"
server_key_version: "encryptionkey@0"
I20260620 15:19:53.164822 15956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 15:19:53.168056 15971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:53.169334 15956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:19:53.169492 15956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "a59e45b9301d4b21b1d72f1f36e4a247"
format_stamp: "Formatted at 2026-06-20 15:19:53 on dist-test-slave-7s64"
server_key: "e178fdaf7efc66a2f7a7c5c9d4255cb2"
server_key_iv: "c3f94ddb942a5f39253b776e69e1e6e3"
server_key_version: "encryptionkey@0"
I20260620 15:19:53.169695 15956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:53.190352 15956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:53.194981 15956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:53.195276 15956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:53.196352 15956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:53.197678 15956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:53.197788 15956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:53.197928 15956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:53.197992 15956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:53.230348 15956 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:44503
I20260620 15:19:53.230432 16083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:44503 every 8 connection(s)
I20260620 15:19:53.231564 15956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:19:53.233247 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 15956
I20260620 15:19:53.233464 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:19:53.234184 389 external_mini_cluster.cc:1428] Setting key cb52d78554d64c88dd8defe3fe0f7698
Jun 20 15:19:53 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:53.254235 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:53.234182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:36065 (local address 127.0.97.126:35917)
0620 15:19:53.235059 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:53.235064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:53.235919 (+ 855us) server_negotiation.cc:407] Connection header received
0620 15:19:53.238552 (+ 2633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:53.238557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:53.238616 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:53.238713 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:53.240733 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:53.241416 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:53.243844 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:53.244223 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:53.244313 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:53.248379 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:53.248384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:53.248387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:53.248422 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:53.251043 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:53.251984 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:53.251988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:53.251990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:53.252107 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:53.252585 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:53.252588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:53.252590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:53.252839 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:53.252945 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:53.253837 (+ 892us) server_negotiation.cc:299] Negotiation successful
0620 15:19:53.253996 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":642,"threads_started":1}
I20260620 15:19:53.255861 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:53.234768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:36065)
0620 15:19:53.235735 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:53.235800 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:53.238279 (+ 2479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:53.238993 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:53.239007 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:53.239772 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:53.240564 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:53.240590 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:53.241568 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:53.241576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:53.243686 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:53.243703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:53.244445 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:53.244454 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:53.244814 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:53.245503 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:53.245529 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:53.248177 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:53.251181 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:53.251192 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:53.251209 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:53.251854 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:53.252220 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:53.252227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:53.252234 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:53.252458 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:53.252972 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:53.252982 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:53.253413 (+ 431us) client_negotiation.cc:769] Sending connection context
0620 15:19:53.253734 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 15:19:53.254298 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":453,"threads_started":1}
I20260620 15:19:53.257539 16084 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35917
I20260620 15:19:53.257831 16084 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:53.258524 16084 heartbeater.cc:511] Master 127.0.97.126:35917 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:19:53.260758 15895 ts_manager.cc:194] Registered new tserver with Master: a59e45b9301d4b21b1d72f1f36e4a247 (127.0.97.65:44503)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260620 15:19:53.262554 15895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:36065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:19:53.288139 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35917
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:53.596889 16090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:53.597209 16090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:53.597304 16090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:53.604259 16090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:53.604351 16090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:53.604457 16090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:19:53.613729 16090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:53.614825 16090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:53.615983 16090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:53.629032 16098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:53.629115 16095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:53.629112 16096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:53.633877 16090 server_base.cc:1034] running on GCE node
I20260620 15:19:53.634462 16090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:53.635299 16090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:53.636536 16090 hybrid_clock.cc:648] HybridClock initialized: now 1781968793636460 us; error 22 us; skew 500 ppm
I20260620 15:19:53.636839 16090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:53 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:53.639734 16090 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:19:53.641196 16090 webserver.cc:460] Webserver started at http://127.0.97.66:33059/ using document root <none> and password file <none>
I20260620 15:19:53.641794 16090 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:53.641885 16090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:53.642124 16090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:53.644850 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "efec529f7a1a44d4b5f155724656c9b4"
format_stamp: "Formatted at 2026-06-20 15:19:53 on dist-test-slave-7s64"
server_key: "48e73ae2670ca0fbcf32b9a941d3d771"
server_key_iv: "db12dcab2cab4412c008550f60f3378f"
server_key_version: "encryptionkey@0"
I20260620 15:19:53.645509 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "efec529f7a1a44d4b5f155724656c9b4"
format_stamp: "Formatted at 2026-06-20 15:19:53 on dist-test-slave-7s64"
server_key: "48e73ae2670ca0fbcf32b9a941d3d771"
server_key_iv: "db12dcab2cab4412c008550f60f3378f"
server_key_version: "encryptionkey@0"
I20260620 15:19:53.650506 16090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:19:53.653828 16105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:53.655076 16090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:19:53.655251 16090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "efec529f7a1a44d4b5f155724656c9b4"
format_stamp: "Formatted at 2026-06-20 15:19:53 on dist-test-slave-7s64"
server_key: "48e73ae2670ca0fbcf32b9a941d3d771"
server_key_iv: "db12dcab2cab4412c008550f60f3378f"
server_key_version: "encryptionkey@0"
I20260620 15:19:53.655443 16090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:53.672899 16090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:53.676736 16090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:53.677019 16090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:53.677820 16090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:53.678977 16090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:53.679046 16090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:53.679127 16090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:53.679178 16090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:53.712080 16090 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:35249
I20260620 15:19:53.712174 16217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:35249 every 8 connection(s)
I20260620 15:19:53.713326 16090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:19:53.717800 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 16090
I20260620 15:19:53.717981 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:19:53.718443 389 external_mini_cluster.cc:1428] Setting key 62cd10c84d268ad1e51893836bf9fd5b
Jun 20 15:19:53 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:53.734898 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:53.715816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:50609 (local address 127.0.97.126:35917)
0620 15:19:53.715998 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:53.716002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:53.717129 (+ 1127us) server_negotiation.cc:407] Connection header received
0620 15:19:53.719096 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:53.719101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:53.719203 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:53.719317 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:53.721345 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:53.722529 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:53.724142 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:53.724916 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:53.725059 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:53.728781 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:53.728787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:53.728790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:53.728828 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:53.731614 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:53.732374 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:53.732382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:53.732395 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:53.732592 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:53.733056 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:53.733062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:53.733065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:53.733354 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:53.733542 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:53.734248 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 15:19:53.734475 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:19:53.735809 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:53.716267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:50609)
0620 15:19:53.716970 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:53.717006 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:53.718813 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:53.719672 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:53.719684 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:53.720480 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:53.721159 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:53.721178 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:53.722724 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:53.722728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:53.723989 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:53.724002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:53.725136 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:53.725142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:53.725340 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:53.726066 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:53.726100 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:53.728477 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:53.731779 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:53.731788 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:53.731802 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:53.732234 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:53.732721 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:53.732726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:53.732730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:53.732924 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:53.733523 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:53.733530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:53.733872 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 15:19:53.734146 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 15:19:53.734651 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":310,"threads_started":1}
I20260620 15:19:53.737356 16218 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35917
I20260620 15:19:53.737674 16218 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:53.738397 16218 heartbeater.cc:511] Master 127.0.97.126:35917 requested a full tablet report, sending...
I20260620 15:19:53.739919 15895 ts_manager.cc:194] Registered new tserver with Master: efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249)
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260620 15:19:53.740895 15895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:50609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:19:53.768116 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35917
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:19:54.084965 16223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:19:54.085215 16223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:19:54.085305 16223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:19:54.092254 16223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:19:54.092378 16223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:19:54.092484 16223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:19:54.101936 16223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:19:54.103117 16223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:19:54.104323 16223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:19:54.117318 16229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:54.117192 16231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:19:54.117445 16228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:19:54.118405 16223 server_base.cc:1034] running on GCE node
I20260620 15:19:54.118916 16223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:19:54.119732 16223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:19:54.120918 16223 hybrid_clock.cc:648] HybridClock initialized: now 1781968794120886 us; error 29 us; skew 500 ppm
I20260620 15:19:54.121167 16223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:19:54 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:19:54.127800 16223 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:19:54.129245 16223 webserver.cc:460] Webserver started at http://127.0.97.67:41521/ using document root <none> and password file <none>
I20260620 15:19:54.129849 16223 fs_manager.cc:359] Metadata directory not provided
I20260620 15:19:54.129942 16223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:19:54.130192 16223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:19:54.133033 16223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "5c42a3142da7467c92998b03ea3b74df"
format_stamp: "Formatted at 2026-06-20 15:19:54 on dist-test-slave-7s64"
server_key: "0f326f949aa78cd46c3668931e639e92"
server_key_iv: "e8d2aa033817eab68ce265fb7c1a8e96"
server_key_version: "encryptionkey@0"
I20260620 15:19:54.133698 16223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "5c42a3142da7467c92998b03ea3b74df"
format_stamp: "Formatted at 2026-06-20 15:19:54 on dist-test-slave-7s64"
server_key: "0f326f949aa78cd46c3668931e639e92"
server_key_iv: "e8d2aa033817eab68ce265fb7c1a8e96"
server_key_version: "encryptionkey@0"
I20260620 15:19:54.138387 16223 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:19:54.141849 16238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:54.143126 16223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:19:54.143309 16223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "5c42a3142da7467c92998b03ea3b74df"
format_stamp: "Formatted at 2026-06-20 15:19:54 on dist-test-slave-7s64"
server_key: "0f326f949aa78cd46c3668931e639e92"
server_key_iv: "e8d2aa033817eab68ce265fb7c1a8e96"
server_key_version: "encryptionkey@0"
I20260620 15:19:54.143504 16223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:19:54.154803 16223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:19:54.158795 16223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:19:54.159093 16223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:19:54.160147 16223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:19:54.161511 16223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:19:54.161580 16223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:54.161769 16223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:19:54.161818 16223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:19:54.198060 16223 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39163
I20260620 15:19:54.198172 16350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39163 every 8 connection(s)
I20260620 15:19:54.199383 16223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:19:54.207931 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 16223
I20260620 15:19:54.208127 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:19:54.208571 389 external_mini_cluster.cc:1428] Setting key 251845beb08da6fe461c42b93449b4b8
Jun 20 15:19:54 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:19:54.219606 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.201967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:56899 (local address 127.0.97.126:35917)
0620 15:19:54.202155 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.202160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.203295 (+ 1135us) server_negotiation.cc:407] Connection header received
0620 15:19:54.205081 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.205086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.205181 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.205272 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:54.206658 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.207393 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.209242 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.209789 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.209930 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.213675 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:54.213687 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:54.213691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:54.213732 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:54.216762 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:54.217460 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:54.217470 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:54.217473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:54.217583 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:54.218000 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:54.218005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:54.218008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:54.218278 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:54.218431 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.219056 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.219264 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:19:54.220427 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.202439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:56899)
0620 15:19:54.203113 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.203164 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:54.204836 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:54.205473 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:54.205481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:54.205953 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:19:54.206490 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.206506 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.207638 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.207643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.209074 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.209090 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.210006 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.210016 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.210258 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.210935 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:19:54.210958 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:19:54.213461 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:19:54.216902 (+ 3441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:54.216910 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:54.216923 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:54.217330 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:54.217692 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:19:54.217700 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:19:54.217705 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:19:54.217878 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:19:54.218404 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:19:54.218411 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:19:54.218710 (+ 299us) client_negotiation.cc:769] Sending connection context
0620 15:19:54.218951 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 15:19:54.219426 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":263,"threads_started":1}
I20260620 15:19:54.221742 16351 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35917
I20260620 15:19:54.222050 16351 heartbeater.cc:464] Registering TS with master...
I20260620 15:19:54.222684 16351 heartbeater.cc:511] Master 127.0.97.126:35917 requested a full tablet report, sending...
I20260620 15:19:54.224169 15895 ts_manager.cc:194] Registered new tserver with Master: 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163)
I20260620 15:19:54.225008 15895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:56899
I20260620 15:19:54.235351 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:19:54.251626 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.239651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.0.97.126:35917)
0620 15:19:54.239814 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.239819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.240238 (+ 419us) server_negotiation.cc:407] Connection header received
0620 15:19:54.240394 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.240397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.240449 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.240592 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:54.241985 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.242666 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.244005 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.244404 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.244559 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.246794 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:54.246799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:54.246802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:54.246829 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:54.249414 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:54.250098 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:54.250102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:54.250104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:54.250221 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:54.250538 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:54.250541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:54.250543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:54.250810 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:54.251010 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.251287 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.251422 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:19:54.255136 15895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:19:54.262135 15895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:19:54.266016 16084 heartbeater.cc:503] Master 127.0.97.126:35917 was elected leader, sending a full tablet report...
I20260620 15:19:54.286851 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.274058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39163 (local address 127.0.0.1:45902)
0620 15:19:54.275353 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.275375 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:54.275515 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:54.277029 (+ 1514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:54.277039 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:54.277087 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:54.277585 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.277600 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.281725 (+ 4125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.281732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.284115 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.284133 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.285966 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.285974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.286163 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.286223 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 15:19:54.286350 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 15:19:54.286510 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"thread_start_us":986,"threads_started":1}
I20260620 15:19:54.286893 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.274693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.0.97.67:39163)
0620 15:19:54.276209 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.276221 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.276260 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:19:54.276373 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.276383 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.276712 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.277845 (+ 1133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:54.278160 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.280270 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.284351 (+ 4081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.285777 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.285910 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.286113 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.286416 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.286571 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1311,"thread_start_us":1093,"threads_started":1}
I20260620 15:19:54.288764 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.273934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:49106)
0620 15:19:54.274608 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.274655 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:54.274815 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:54.281419 (+ 6604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:54.281423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:54.281449 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:54.281944 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.281960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.283839 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.283844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.285921 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.285935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.287887 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.287893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.288048 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.288080 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:19:54.288178 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:19:54.288306 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":44,"thread_start_us":288,"threads_started":1}
I20260620 15:19:54.289317 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.276858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.0.97.66:35249)
0620 15:19:54.281154 (+ 4296us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.281161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.281792 (+ 631us) server_negotiation.cc:407] Connection header received
0620 15:19:54.281891 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.281900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.282100 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.282291 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:54.282849 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.283868 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.286528 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.287785 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.287906 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.288082 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.288845 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.289001 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4120,"thread_start_us":3999,"threads_started":1}
I20260620 15:19:54.289381 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.274137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.0.97.65:44503)
0620 15:19:54.280886 (+ 6749us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.280895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.280926 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:19:54.281017 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.281023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.281245 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.281424 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:54.282149 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.283699 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.286099 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.287686 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.287807 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.288010 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.288956 (+ 946us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.289091 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6589,"thread_start_us":6426,"threads_started":1}
I20260620 15:19:54.289776 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.276560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:35249 (local address 127.0.0.1:54388)
0620 15:19:54.281506 (+ 4946us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.281524 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:54.281654 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:54.282280 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:54.282284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:54.282308 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:54.282700 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.282708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.284013 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.284019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.286305 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.286343 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.288228 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.288233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.288378 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.288424 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 15:19:54.288527 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:19:54.288644 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4826,"mutex_wait_us":22,"thread_start_us":4703,"threads_started":1}
I20260620 15:19:54.291034 16286 tablet_service.cc:1467] Processing CreateTablet for tablet 03beaa5529ce40e4a93370abb09c7745 (DEFAULT_TABLE table=test-table [id=db09592c90e142eb8c2cc8e52f075a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:54.292663 16286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03beaa5529ce40e4a93370abb09c7745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:54.292747 16019 tablet_service.cc:1467] Processing CreateTablet for tablet 03beaa5529ce40e4a93370abb09c7745 (DEFAULT_TABLE table=test-table [id=db09592c90e142eb8c2cc8e52f075a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:54.293337 16153 tablet_service.cc:1467] Processing CreateTablet for tablet 03beaa5529ce40e4a93370abb09c7745 (DEFAULT_TABLE table=test-table [id=db09592c90e142eb8c2cc8e52f075a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:19:54.294097 16019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03beaa5529ce40e4a93370abb09c7745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:54.294588 16153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03beaa5529ce40e4a93370abb09c7745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:19:54.304384 16369 tablet_bootstrap.cc:492] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: Bootstrap starting.
I20260620 15:19:54.306824 16370 tablet_bootstrap.cc:492] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: Bootstrap starting.
I20260620 15:19:54.308326 16371 tablet_bootstrap.cc:492] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: Bootstrap starting.
I20260620 15:19:54.308939 16369 tablet_bootstrap.cc:654] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:54.310129 16370 tablet_bootstrap.cc:654] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:54.310197 16369 log.cc:824] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:54.311506 16371 tablet_bootstrap.cc:654] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: Neither blocks nor log segments found. Creating new log.
I20260620 15:19:54.311633 16370 log.cc:824] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:54.312842 16371 log.cc:824] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:19:54.314157 16369 tablet_bootstrap.cc:492] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: No bootstrap required, opened a new log
I20260620 15:19:54.314419 16369 ts_tablet_manager.cc:1397] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 15:19:54.314487 16370 tablet_bootstrap.cc:492] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: No bootstrap required, opened a new log
I20260620 15:19:54.314738 16370 ts_tablet_manager.cc:1397] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:19:54.315999 16371 tablet_bootstrap.cc:492] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: No bootstrap required, opened a new log
I20260620 15:19:54.316231 16371 ts_tablet_manager.cc:1397] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:19:54.321610 16370 raft_consensus.cc:348] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } } peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } }
I20260620 15:19:54.321955 16370 raft_consensus.cc:374] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:54.322053 16370 raft_consensus.cc:729] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c42a3142da7467c92998b03ea3b74df, State: Initialized, Role: FOLLOWER
I20260620 15:19:54.322639 16370 consensus_queue.cc:260] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } } peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } }
I20260620 15:19:54.323736 16351 heartbeater.cc:503] Master 127.0.97.126:35917 was elected leader, sending a full tablet report...
I20260620 15:19:54.324374 16370 ts_tablet_manager.cc:1428] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 15:19:54.324764 16369 raft_consensus.cc:348] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } } peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } }
I20260620 15:19:54.325166 16369 raft_consensus.cc:374] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:54.325271 16369 raft_consensus.cc:729] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a59e45b9301d4b21b1d72f1f36e4a247, State: Initialized, Role: FOLLOWER
I20260620 15:19:54.325855 16369 consensus_queue.cc:260] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } } peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } }
I20260620 15:19:54.326821 16371 raft_consensus.cc:348] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } } peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } }
I20260620 15:19:54.327265 16371 raft_consensus.cc:374] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:19:54.327401 16371 raft_consensus.cc:729] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efec529f7a1a44d4b5f155724656c9b4, State: Initialized, Role: FOLLOWER
I20260620 15:19:54.327553 16369 ts_tablet_manager.cc:1428] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 15:19:54.328095 16371 consensus_queue.cc:260] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } } peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } }
I20260620 15:19:54.329214 16218 heartbeater.cc:503] Master 127.0.97.126:35917 was elected leader, sending a full tablet report...
I20260620 15:19:54.329895 16371 ts_tablet_manager.cc:1428] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260620 15:19:54.378374 16376 raft_consensus.cc:482] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:19:54.378690 16376 raft_consensus.cc:504] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } } peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } }
I20260620 15:19:54.380368 16376 leader_election.cc:288] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163), efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249)
I20260620 15:19:54.389792 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.380817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34495 (local address 127.0.97.67:39163)
0620 15:19:54.381047 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.381056 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.381546 (+ 490us) server_negotiation.cc:407] Connection header received
0620 15:19:54.381826 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.381835 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.381932 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.382009 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:54.382920 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.384083 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.387310 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.388581 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.388718 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.388819 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.389393 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.389519 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:19:54.389792 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.381724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34893 (local address 127.0.97.66:35249)
0620 15:19:54.381926 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.381931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.382146 (+ 215us) server_negotiation.cc:407] Connection header received
0620 15:19:54.382313 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.382318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.382375 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.382448 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:19:54.383127 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.384154 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.387304 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.388576 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.388718 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.388815 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.389393 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.389521 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:19:54.390048 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.381411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:35249 (local address 127.0.97.65:34893)
0620 15:19:54.382044 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.382064 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:54.382192 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:54.382505 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:54.382510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:54.382531 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:54.382980 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.382989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.384284 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.384292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.387089 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.387104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.388858 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.388863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.389098 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.389254 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 15:19:54.389351 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 15:19:54.389565 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":302,"thread_start_us":283,"threads_started":1}
I20260620 15:19:54.390072 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.380610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39163 (local address 127.0.97.65:34495)
0620 15:19:54.381413 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.381436 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:19:54.381602 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:19:54.382259 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:19:54.382267 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:19:54.382303 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:19:54.382735 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.382755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.384256 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.384263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.387091 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:19:54.387104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.388831 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.388843 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:19:54.389095 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.389250 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 15:19:54.389357 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 15:19:54.389565 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":403,"thread_start_us":447,"threads_started":1}
I20260620 15:19:54.391043 16306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03beaa5529ce40e4a93370abb09c7745" candidate_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c42a3142da7467c92998b03ea3b74df" is_pre_election: true
I20260620 15:19:54.391043 16173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03beaa5529ce40e4a93370abb09c7745" candidate_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efec529f7a1a44d4b5f155724656c9b4" is_pre_election: true
I20260620 15:19:54.391569 16306 raft_consensus.cc:2454] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a59e45b9301d4b21b1d72f1f36e4a247 in term 0.
I20260620 15:19:54.391569 16173 raft_consensus.cc:2454] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a59e45b9301d4b21b1d72f1f36e4a247 in term 0.
I20260620 15:19:54.392205 15972 leader_election.cc:302] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c42a3142da7467c92998b03ea3b74df, a59e45b9301d4b21b1d72f1f36e4a247; no voters:
I20260620 15:19:54.392627 16376 raft_consensus.cc:2781] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:19:54.392802 16376 raft_consensus.cc:482] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:19:54.392921 16376 raft_consensus.cc:3037] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:54.395171 16376 raft_consensus.cc:504] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } } peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } }
I20260620 15:19:54.395741 16376 leader_election.cc:288] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 1 election: Requested vote from peers 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163), efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249)
I20260620 15:19:54.396193 16306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03beaa5529ce40e4a93370abb09c7745" candidate_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c42a3142da7467c92998b03ea3b74df"
I20260620 15:19:54.396299 16173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03beaa5529ce40e4a93370abb09c7745" candidate_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efec529f7a1a44d4b5f155724656c9b4"
I20260620 15:19:54.396386 16306 raft_consensus.cc:3037] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:54.396562 16173 raft_consensus.cc:3037] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:19:54.398893 16306 raft_consensus.cc:2454] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a59e45b9301d4b21b1d72f1f36e4a247 in term 1.
I20260620 15:19:54.399000 16173 raft_consensus.cc:2454] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a59e45b9301d4b21b1d72f1f36e4a247 in term 1.
I20260620 15:19:54.399335 15972 leader_election.cc:302] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c42a3142da7467c92998b03ea3b74df, a59e45b9301d4b21b1d72f1f36e4a247; no voters:
I20260620 15:19:54.399626 16376 raft_consensus.cc:2781] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:19:54.400044 16376 raft_consensus.cc:686] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 LEADER]: Becoming Leader. State: Replica: a59e45b9301d4b21b1d72f1f36e4a247, State: Running, Role: LEADER
I20260620 15:19:54.400555 16376 consensus_queue.cc:237] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } } peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } }
I20260620 15:19:54.404528 15895 catalog_manager.cc:5495] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 reported cstate change: term changed from 0 to 1, leader changed from <none> to a59e45b9301d4b21b1d72f1f36e4a247 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } health_report { overall_health: UNKNOWN } } }
I20260620 15:19:54.438591 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.433233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.0.97.65:44503)
0620 15:19:54.433445 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.433450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.433468 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:19:54.433640 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.433644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.433729 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.433875 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:19:54.434245 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.434941 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.436388 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.436888 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.437016 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.437497 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:19:54.438044 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:19:54.438213 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.438262 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.438345 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:19:54.446034 16306 raft_consensus.cc:1261] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df [term 1 FOLLOWER]: Refusing update from remote peer a59e45b9301d4b21b1d72f1f36e4a247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:19:54.446033 16173 raft_consensus.cc:1261] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4 [term 1 FOLLOWER]: Refusing update from remote peer a59e45b9301d4b21b1d72f1f36e4a247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:19:54.446934 16376 consensus_queue.cc:1035] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:19:54.447283 16380 consensus_queue.cc:1035] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:19:54.451051 16352 tablet.cc:2369] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:19:54.457635 16383 mvcc.cc:205] Tried to move back new op lower bound from 7298944182041665536 to 7298944181867921408. Current Snapshot: MvccSnapshot[applied={T|T < 7298944182041665536}]
I20260620 15:19:54.469375 15895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:19:54.469617 15895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:19:54.472790 15895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Sending DeleteTablet for 3 replicas of tablet 03beaa5529ce40e4a93370abb09c7745
I20260620 15:19:54.473860 16019 tablet_service.cc:1514] Processing DeleteTablet for tablet 03beaa5529ce40e4a93370abb09c7745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:54 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49106
I20260620 15:19:54.474054 16286 tablet_service.cc:1514] Processing DeleteTablet for tablet 03beaa5529ce40e4a93370abb09c7745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:54 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:45902
I20260620 15:19:54.474093 16153 tablet_service.cc:1514] Processing DeleteTablet for tablet 03beaa5529ce40e4a93370abb09c7745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:19:54 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:54388
I20260620 15:19:54.479555 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 15956
I20260620 15:19:54.483834 16394 tablet_replica.cc:330] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: stopping tablet replica
I20260620 15:19:54.484968 16394 raft_consensus.cc:2229] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:19:54.486173 16394 raft_consensus.cc:2258] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:54.490485 16394 ts_tablet_manager.cc:1905] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:54.494768 16392 tablet_replica.cc:330] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: stopping tablet replica
I20260620 15:19:54.496800 16393 tablet_replica.cc:330] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: stopping tablet replica
I20260620 15:19:54.496925 16392 raft_consensus.cc:2229] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:19:54.497259 16393 raft_consensus.cc:2229] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:19:54.497604 16392 raft_consensus.cc:2258] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:54.497839 16393 raft_consensus.cc:2258] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:19:54.499928 16393 ts_tablet_manager.cc:1905] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:54.501389 16394 ts_tablet_manager.cc:1918] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:54.501579 16394 log.cc:1196] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/03beaa5529ce40e4a93370abb09c7745
I20260620 15:19:54.502148 16394 ts_tablet_manager.cc:1939] T 03beaa5529ce40e4a93370abb09c7745 P 5c42a3142da7467c92998b03ea3b74df: Deleting consensus metadata
Jun 20 15:19:54 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:19:54.503957 15879 catalog_manager.cc:4841] TS 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163): tablet 03beaa5529ce40e4a93370abb09c7745 (table test-table [id=db09592c90e142eb8c2cc8e52f075a02]) successfully deleted
I20260620 15:19:54.506487 16393 ts_tablet_manager.cc:1918] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:54.506702 16393 log.cc:1196] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/03beaa5529ce40e4a93370abb09c7745
I20260620 15:19:54.507056 16392 ts_tablet_manager.cc:1905] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:19:54.507364 16393 ts_tablet_manager.cc:1939] T 03beaa5529ce40e4a93370abb09c7745 P efec529f7a1a44d4b5f155724656c9b4: Deleting consensus metadata
I20260620 15:19:54.509054 15879 catalog_manager.cc:4841] TS efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249): tablet 03beaa5529ce40e4a93370abb09c7745 (table test-table [id=db09592c90e142eb8c2cc8e52f075a02]) successfully deleted
I20260620 15:19:54.512683 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.480742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.0.97.65:44503)
0620 15:19:54.480970 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.480980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.491420 (+ 10440us) server_negotiation.cc:407] Connection header received
0620 15:19:54.491892 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.491897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.491972 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.492045 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:54.494955 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.496165 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.498246 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.498943 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.499041 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.504685 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:54.504699 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:54.504721 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:54.504769 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:54.509309 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:54.510027 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:54.510037 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:54.510041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:54.510204 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:54.510728 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:54.510733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:54.510736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:54.511115 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:54.511471 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.511879 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.512115 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:19:54.513387 16079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:54.609462 16392 ts_tablet_manager.cc:1918] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:19:54.609916 16392 log.cc:1196] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/03beaa5529ce40e4a93370abb09c7745
I20260620 15:19:54.610181 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 15956
W20260620 15:19:54.628393 15881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:44503 (error 108)
I20260620 15:19:54.629096 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 16090
I20260620 15:19:54.630509 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.629910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:49132)
0620 15:19:54.630191 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.630305 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
W20260620 15:19:54.630781 15881 catalog_manager.cc:4584] TS a59e45b9301d4b21b1d72f1f36e4a247 (127.0.97.65:44503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03beaa5529ce40e4a93370abb09c7745: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Jun 20 15:19:54 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:19:54.644078 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.629893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.0.97.66:35249)
0620 15:19:54.630077 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.630081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.630160 (+ 79us) server_negotiation.cc:407] Connection header received
0620 15:19:54.630348 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.630352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.630421 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.630500 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:54.631909 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.632813 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.634097 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.634471 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.634564 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.638749 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:54.638762 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:54.638776 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:54.638805 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:54.641694 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:54.642290 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:54.642296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:54.642299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:54.642466 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:54.642772 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:54.642776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:54.642777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:54.643119 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:54.643346 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.643575 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.643723 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:19:54.644790 16213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:54.684167 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.683682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:49146)
0620 15:19:54.683903 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.684009 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:19:54.685153 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.684803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:49148)
0620 15:19:54.684944 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.685030 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 15:19:54.743440 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 16090
I20260620 15:19:54.745873 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.745312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:49152)
0620 15:19:54.745550 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.745658 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 15:19:54.746899 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.746508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:49158)
0620 15:19:54.746681 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.746782 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:19:54.760450 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 16223
Jun 20 15:19:54 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:19:54.774552 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.761224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.0.97.67:39163)
0620 15:19:54.761395 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:19:54.761400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:19:54.761418 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:19:54.761568 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:19:54.761572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:19:54.761641 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:19:54.761743 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:19:54.762954 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.763697 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.765125 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:19:54.765515 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:19:54.765611 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:19:54.769808 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:19:54.769822 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:19:54.769837 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:19:54.769870 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:19:54.772272 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:54.772951 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:54.772958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:54.772960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:54.773081 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:19:54.773388 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:19:54.773391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:19:54.773393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:19:54.773696 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:19:54.773897 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 15:19:54.774173 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 15:19:54.774325 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:19:54.775166 16346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:54.827641 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.827106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:49172)
0620 15:19:54.827376 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.827484 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260620 15:19:54.828747 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:19:54.828317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:49182)
0620 15:19:54.828581 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 15:19:54.828653 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 15:19:54.863994 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 16223
I20260620 15:19:54.882254 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 15863
I20260620 15:19:54.883430 15925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:19:54.994617 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 15863
I20260620 15:20:15.015899 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35917
--webserver_interface=127.0.97.126
--webserver_port=41943
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:20:15.336611 16401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:15.336884 16401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:15.337028 16401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:15.344154 16401 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:20:15.344236 16401 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:20:15.344306 16401 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:20:15.344383 16401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:20:15.344476 16401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:15.344611 16401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:15.344691 16401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:20:15.344767 16401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:20:15.354072 16401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=41943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:15.355258 16401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:15.356567 16401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:15.378419 16407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:15.378877 16408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:15.378589 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:15.379175 16401 server_base.cc:1034] running on GCE node
I20260620 15:20:15.380079 16401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:15.381587 16401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:15.382843 16401 hybrid_clock.cc:648] HybridClock initialized: now 1781968815382761 us; error 61 us; skew 500 ppm
I20260620 15:20:15.383198 16401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:15 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:15.390455 16401 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:20:15.392072 16401 webserver.cc:460] Webserver started at http://127.0.97.126:41943/ using document root <none> and password file <none>
I20260620 15:20:15.392908 16401 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:15.393064 16401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:15.398620 16401 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 15:20:15.401684 16417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:15.403021 16401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:20:15.403208 16401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "2ca71167a4574b0591d4a58e89ce4ba0"
format_stamp: "Formatted at 2026-06-20 15:19:52 on dist-test-slave-7s64"
server_key: "7f31729cc8c6f72f47d92d9237b6cbeb"
server_key_iv: "ddc442ee3361b1818b4ed91a45d3f18d"
server_key_version: "encryptionkey@0"
I20260620 15:20:15.404233 16401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:15.426054 16401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:15.430341 16401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:15.430611 16401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:15.448998 16401 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35917
I20260620 15:20:15.449043 16468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35917 every 8 connection(s)
I20260620 15:20:15.450428 16401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:20:15.456784 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 16401
I20260620 15:20:15.457252 16469 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:20:15.457641 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:44503
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:35917
--webserver_port=40089
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:20:15.460616 16469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Bootstrap starting.
I20260620 15:20:15.488448 16469 log.cc:824] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Log is configured to *not* fsync() on all Append() calls
I20260620 15:20:15.501298 16469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:20:15.501829 16469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Bootstrap complete.
I20260620 15:20:15.511319 16469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:20:15.512228 16469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca71167a4574b0591d4a58e89ce4ba0, State: Initialized, Role: FOLLOWER
I20260620 15:20:15.512892 16469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:20:15.513145 16469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:20:15.513275 16469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:20:15.513465 16469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:20:15.517266 16469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:20:15.517774 16469 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca71167a4574b0591d4a58e89ce4ba0; no voters:
I20260620 15:20:15.518389 16469 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:20:15.518580 16473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:20:15.519696 16473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 2 LEADER]: Becoming Leader. State: Replica: 2ca71167a4574b0591d4a58e89ce4ba0, State: Running, Role: LEADER
I20260620 15:20:15.520356 16469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:20:15.520313 16473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:20:15.523372 16475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca71167a4574b0591d4a58e89ce4ba0. Latest consensus state: current_term: 2 leader_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } } }
I20260620 15:20:15.523782 16475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: This master's current role is: LEADER
I20260620 15:20:15.523245 16474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } } }
I20260620 15:20:15.524830 16474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: This master's current role is: LEADER
I20260620 15:20:15.533002 16478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:20:15.541236 16478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:20:15.542626 16478 catalog_manager.cc:1259] Loaded cluster ID: 8466b4886aee46158e337fb5c47e956a
I20260620 15:20:15.542783 16478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:20:15.546818 16478 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:20:15.548399 16478 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Loaded TSK: 0
I20260620 15:20:15.567648 16490 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:20:15.582078 16478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Generated new TSK 1
I20260620 15:20:15.582438 16478 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:20:15.809765 16471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:15.810009 16471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:15.810145 16471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:15.817575 16471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:15.817670 16471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:15.817832 16471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:20:15.827219 16471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:44503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=40089
--tserver_master_addrs=127.0.97.126:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:15.828367 16471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:15.829665 16471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:15.843956 16497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:15.844453 16496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:15.844565 16499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:15.845767 16471 server_base.cc:1034] running on GCE node
I20260620 15:20:15.846379 16471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:15.847272 16471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:15.848536 16471 hybrid_clock.cc:648] HybridClock initialized: now 1781968815848436 us; error 51 us; skew 500 ppm
I20260620 15:20:15.848845 16471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:15 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:15.852063 16471 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:20:15.853565 16471 webserver.cc:460] Webserver started at http://127.0.97.65:40089/ using document root <none> and password file <none>
I20260620 15:20:15.854214 16471 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:15.854367 16471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:15.859421 16471 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:20:15.862140 16506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:15.863461 16471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:20:15.863615 16471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "a59e45b9301d4b21b1d72f1f36e4a247"
format_stamp: "Formatted at 2026-06-20 15:19:53 on dist-test-slave-7s64"
server_key: "e178fdaf7efc66a2f7a7c5c9d4255cb2"
server_key_iv: "c3f94ddb942a5f39253b776e69e1e6e3"
server_key_version: "encryptionkey@0"
I20260620 15:20:15.864534 16471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:15.879119 16471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:15.883176 16471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:15.883440 16471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:15.884436 16471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:20:15.886281 16513 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 15:20:15.891259 16513 ts_tablet_manager.cc:1852] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 15:20:15.892186 16513 ts_tablet_manager.cc:1905] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:20:15.899025 16513 ts_tablet_manager.cc:1918] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:20:15.899242 16513 ts_tablet_manager.cc:1939] T 03beaa5529ce40e4a93370abb09c7745 P a59e45b9301d4b21b1d72f1f36e4a247: Deleting consensus metadata
I20260620 15:20:15.900043 16471 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 15:20:15.900130 16471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.001s
I20260620 15:20:15.900214 16471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:20:15.900274 16471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:15.933287 16471 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:44503
I20260620 15:20:15.933347 16619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:44503 every 8 connection(s)
I20260620 15:20:15.934551 16471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:20:15.943512 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 16471
I20260620 15:20:15.944092 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:35249
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:35917
--webserver_port=33059
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:20:15 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:20:15.956969 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:15.937342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:32981 (local address 127.0.97.126:35917)
0620 15:20:15.937971 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:15.938042 (+ 71us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:15.938460 (+ 418us) server_negotiation.cc:407] Connection header received
0620 15:20:15.940733 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:15.940746 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:15.941042 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:15.941320 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:15.942648 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:15.943911 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:15.945532 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:15.946153 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:15.946337 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:15.949973 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:15.949980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:15.949992 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:15.950026 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:15.952911 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:15.953596 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:15.953608 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:15.953613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:15.953761 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:15.954148 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:15.954155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:15.954158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:15.954566 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:15.954769 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:15.955459 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 15:20:15.955818 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":212,"threads_started":1}
I20260620 15:20:15.956961 16623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:15.937465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:32981)
0620 15:20:15.938287 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:15.938322 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:15.940201 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:15.941450 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:15.941458 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:15.941989 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:20:15.942485 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:15.942501 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:15.944061 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:15.944066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:15.945387 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:15.945398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:15.946363 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:15.946370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:15.946619 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:15.947263 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:20:15.947283 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:20:15.949704 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:20:15.953056 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:15.953064 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:15.953076 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:15.953465 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:15.953871 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:15.953877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:15.953880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:15.954042 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:15.954703 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:20:15.954711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:20:15.955014 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 15:20:15.955257 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 15:20:15.955730 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":314,"threads_started":1}
I20260620 15:20:15.959993 16620 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35917
I20260620 15:20:15.960332 16620 heartbeater.cc:464] Registering TS with master...
I20260620 15:20:15.961218 16620 heartbeater.cc:511] Master 127.0.97.126:35917 requested a full tablet report, sending...
I20260620 15:20:15.963775 16434 ts_manager.cc:194] Registered new tserver with Master: a59e45b9301d4b21b1d72f1f36e4a247 (127.0.97.65:44503)
I20260620 15:20:15.966223 16434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:32981
W20260620 15:20:16.262944 16624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:16.263199 16624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:16.263295 16624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:16.270160 16624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:16.270257 16624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:16.270361 16624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:20:16.280845 16624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:35249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=33059
--tserver_master_addrs=127.0.97.126:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:16.281934 16624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:16.283150 16624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:16.295698 16629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:16.295692 16630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:16.295549 16632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:16.296643 16624 server_base.cc:1034] running on GCE node
I20260620 15:20:16.297186 16624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:16.298062 16624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:16.299254 16624 hybrid_clock.cc:648] HybridClock initialized: now 1781968816299207 us; error 44 us; skew 500 ppm
I20260620 15:20:16.299544 16624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:16 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:16.302557 16624 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:20:16.303982 16624 webserver.cc:460] Webserver started at http://127.0.97.66:33059/ using document root <none> and password file <none>
I20260620 15:20:16.304710 16624 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:16.304843 16624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:16.309851 16624 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 15:20:16.312300 16639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:16.313674 16624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:20:16.313851 16624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "efec529f7a1a44d4b5f155724656c9b4"
format_stamp: "Formatted at 2026-06-20 15:19:53 on dist-test-slave-7s64"
server_key: "48e73ae2670ca0fbcf32b9a941d3d771"
server_key_iv: "db12dcab2cab4412c008550f60f3378f"
server_key_version: "encryptionkey@0"
I20260620 15:20:16.314761 16624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:16.339394 16624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:16.343080 16624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:16.343328 16624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:16.344180 16624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:20:16.345427 16624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:20:16.345556 16624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:16.345667 16624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:20:16.345757 16624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:16.378752 16624 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:35249
I20260620 15:20:16.378832 16751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:35249 every 8 connection(s)
I20260620 15:20:16.380071 16624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:20:16.386137 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 16624
I20260620 15:20:16.386756 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:39163
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:35917
--webserver_port=41521
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:20:16 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:20:16.399403 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.382496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:48759 (local address 127.0.97.126:35917)
0620 15:20:16.382680 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:16.382684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:16.383736 (+ 1052us) server_negotiation.cc:407] Connection header received
0620 15:20:16.385582 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:16.385588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:16.385681 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:16.385795 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:16.387364 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.388064 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.389631 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.390052 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.390181 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.393676 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:16.393684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:16.393692 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:16.393740 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:16.396321 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:16.397232 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:16.397236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:16.397239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:16.397352 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:16.397857 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:16.397861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:16.397863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:16.398110 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:16.398216 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:16.399006 (+ 790us) server_negotiation.cc:299] Negotiation successful
0620 15:20:16.399166 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:20:16.400858 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.382894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:48759)
0620 15:20:16.383578 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:16.383613 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:16.385328 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:16.386033 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:16.386041 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:16.386557 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:20:16.387200 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:16.387224 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.388222 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.388226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:16.389478 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:16.389489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.390268 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.390279 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:16.390492 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.391123 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:20:16.391143 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:20:16.393444 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:20:16.396459 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:16.396469 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:16.396485 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:16.397114 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:16.397454 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:16.397461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:16.397467 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:16.397701 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:16.398239 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:20:16.398249 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:20:16.398632 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 15:20:16.398923 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 15:20:16.399465 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":284,"threads_started":1}
I20260620 15:20:16.402346 16752 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35917
I20260620 15:20:16.402662 16752 heartbeater.cc:464] Registering TS with master...
I20260620 15:20:16.403348 16752 heartbeater.cc:511] Master 127.0.97.126:35917 requested a full tablet report, sending...
I20260620 15:20:16.405076 16434 ts_manager.cc:194] Registered new tserver with Master: efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249)
I20260620 15:20:16.406375 16434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:48759
W20260620 15:20:16.704192 16755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:16.704439 16755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:16.704604 16755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:16.711755 16755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:16.711845 16755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:16.712016 16755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:20:16.721683 16755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:39163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=41521
--tserver_master_addrs=127.0.97.126:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:16.722791 16755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:16.724011 16755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:16.737161 16760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:16.737195 16761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:16.737277 16763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:16.737882 16755 server_base.cc:1034] running on GCE node
I20260620 15:20:16.738502 16755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:16.739377 16755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:16.740592 16755 hybrid_clock.cc:648] HybridClock initialized: now 1781968816740526 us; error 48 us; skew 500 ppm
I20260620 15:20:16.740901 16755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:16 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:16.743955 16755 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:20:16.745381 16755 webserver.cc:460] Webserver started at http://127.0.97.67:41521/ using document root <none> and password file <none>
I20260620 15:20:16.746033 16755 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:16.746197 16755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:16.751195 16755 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:20:16.753722 16770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:16.754961 16755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:20:16.755141 16755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "5c42a3142da7467c92998b03ea3b74df"
format_stamp: "Formatted at 2026-06-20 15:19:54 on dist-test-slave-7s64"
server_key: "0f326f949aa78cd46c3668931e639e92"
server_key_iv: "e8d2aa033817eab68ce265fb7c1a8e96"
server_key_version: "encryptionkey@0"
I20260620 15:20:16.756057 16755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:16.777908 16755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:16.781626 16755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:16.781879 16755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:16.782840 16755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:20:16.784099 16755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:20:16.784228 16755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:16.784355 16755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:20:16.784442 16755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:16.817664 16755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39163
I20260620 15:20:16.817754 16882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39163 every 8 connection(s)
I20260620 15:20:16.819063 16755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:20:16.819746 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 16755
Jun 20 15:20:16 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:20:16.838361 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.821047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.0.97.126:35917)
0620 15:20:16.821247 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:16.821252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:16.824754 (+ 3502us) server_negotiation.cc:407] Connection header received
0620 15:20:16.824951 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:16.824957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:16.825030 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:16.825132 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:16.826640 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.827416 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.828833 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.829265 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.829404 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.832142 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:16.832148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:16.832152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:16.832194 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:16.835516 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:16.836187 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:16.836192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:16.836194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:16.836339 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:16.836783 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:16.836790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:16.836797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:16.837108 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:16.837452 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:16.837757 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 15:20:16.837992 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"spinlock_wait_cycles":3328}
I20260620 15:20:16.839481 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.822026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:57031 (local address 127.0.97.126:35917)
0620 15:20:16.822444 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:16.822448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:16.823696 (+ 1248us) server_negotiation.cc:407] Connection header received
0620 15:20:16.825579 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:16.825585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:16.825654 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:16.825774 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:16.827165 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.827941 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.829471 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.829907 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.830032 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.833299 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:16.833304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:16.833307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:16.833345 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:16.836224 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:16.837049 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:16.837056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:16.837064 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:16.837223 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:16.837635 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:16.837640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:16.837643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:16.837967 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:16.838100 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:16.838905 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 15:20:16.839093 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"spinlock_wait_cycles":2816,"thread_start_us":233,"threads_started":1}
I20260620 15:20:16.840310 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.822780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:57031)
0620 15:20:16.823553 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:16.823592 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:16.825322 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:16.826019 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:16.826026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:16.826462 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:20:16.826990 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:16.827005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.828088 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.828100 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:16.829323 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:16.829335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.830136 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.830143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:16.830358 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.830941 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:20:16.830965 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:20:16.833043 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:20:16.836364 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:16.836371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:16.836384 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:16.836932 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:16.837337 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:16.837344 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:16.837348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:16.837520 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:16.838079 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:20:16.838086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:20:16.838530 (+ 444us) client_negotiation.cc:769] Sending connection context
0620 15:20:16.838813 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 15:20:16.839268 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":335,"threads_started":1}
I20260620 15:20:16.841670 16883 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35917
I20260620 15:20:16.841955 16883 heartbeater.cc:464] Registering TS with master...
I20260620 15:20:16.842607 16883 heartbeater.cc:511] Master 127.0.97.126:35917 requested a full tablet report, sending...
I20260620 15:20:16.843890 16434 ts_manager.cc:194] Registered new tserver with Master: 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163)
I20260620 15:20:16.844748 16434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:57031
I20260620 15:20:16.853070 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:20:16.859031 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.853907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.0.97.126:35917)
0620 15:20:16.854105 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:16.854113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:16.854511 (+ 398us) server_negotiation.cc:407] Connection header received
0620 15:20:16.854665 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:16.854668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:16.854792 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:16.854900 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:20:16.855278 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.855943 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.857459 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.857843 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.857934 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.858350 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:20:16.858567 (+ 217us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 15:20:16.858820 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53004: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 15:20:16.859169 16886 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53004: token expired
W20260620 15:20:16.859771 16354 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.126:35917)
I20260620 15:20:16.872200 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.860162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.0.97.126:35917)
0620 15:20:16.860337 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:16.860340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:16.860354 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:20:16.860458 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:16.860461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:16.860552 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:16.860622 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:16.861959 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.862634 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.864014 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.864412 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.864601 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.867033 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:16.867037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:16.867041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:16.867070 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:16.869804 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:16.870423 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:16.870427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:16.870429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:16.870543 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:16.870878 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:16.870881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:16.870883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:16.871176 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:16.871363 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:16.871697 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 15:20:16.871873 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:20:16.875597 16434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:20:16.883127 16434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:20:16.916736 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.895697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54730 (local address 127.0.97.65:44503)
0620 15:20:16.906644 (+ 10947us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:16.906653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:16.906684 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:20:16.906788 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:16.906796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:16.907129 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:16.907727 (+ 598us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:20:16.908984 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.910630 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.914343 (+ 3713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.915513 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.915668 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.915877 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:16.916170 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 15:20:16.916349 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10781,"thread_start_us":11012,"threads_started":1}
I20260620 15:20:16.916922 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.895322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:54730)
0620 15:20:16.906505 (+ 11183us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:16.906526 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:16.906643 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:16.908064 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:16.908073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:16.908106 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:20:16.908756 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:16.908776 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.910798 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.910802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:16.914189 (+ 3387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:16.914202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.915717 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.915723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:16.915950 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.915987 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:20:16.916114 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 15:20:16.916348 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11048,"mutex_wait_us":4,"thread_start_us":18050,"threads_started":1}
I20260620 15:20:16.916932 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.894695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39163 (local address 127.0.0.1:40166)
0620 15:20:16.895429 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:16.895480 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:16.895654 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:16.909219 (+ 13565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:16.909223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:16.909249 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:20:16.909612 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:16.909624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.911762 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.911768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:16.914038 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:16.914053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.915824 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.915830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:16.916018 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.916052 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:20:16.916115 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 15:20:16.916351 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":50,"thread_start_us":345,"threads_started":1}
I20260620 15:20:16.918278 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.895002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.0.97.67:39163)
0620 15:20:16.908348 (+ 13346us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:16.908358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:16.908392 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:20:16.908525 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:16.908532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:16.909007 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:16.909799 (+ 792us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:20:16.910075 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.911597 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.914199 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.915604 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.917448 (+ 1844us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.917687 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:16.917752 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 15:20:16.917907 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13142,"thread_start_us":13614,"threads_started":1}
I20260620 15:20:16.919266 16555 tablet_service.cc:1467] Processing CreateTablet for tablet e94109dea54f47bb9d2eebc183ac7c26 (DEFAULT_TABLE table=test-table [id=7a817b6cb18c434e90140872a680a8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:20:16.920783 16555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94109dea54f47bb9d2eebc183ac7c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:20:16.921690 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.896609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.0.97.66:35249)
0620 15:20:16.912953 (+ 16344us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:16.912965 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:16.913796 (+ 831us) server_negotiation.cc:407] Connection header received
0620 15:20:16.913989 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:16.913997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:16.914232 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:16.914422 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:20:16.914959 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.916807 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.919258 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.920690 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.920846 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.921019 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:16.921248 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 15:20:16.921403 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16135,"thread_start_us":16362,"threads_started":1}
I20260620 15:20:16.921695 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:16.895872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:35249 (local address 127.0.0.1:49198)
0620 15:20:16.913661 (+ 17789us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:16.913676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:16.913825 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:16.914416 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:16.914420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:16.914447 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:20:16.914802 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:16.914824 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.916968 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.916976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:16.919094 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:16.919108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:16.920884 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:16.920890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:16.921069 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:16.921105 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:20:16.921239 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 15:20:16.921353 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17661,"thread_start_us":10325,"threads_started":1}
I20260620 15:20:16.922111 16818 tablet_service.cc:1467] Processing CreateTablet for tablet e94109dea54f47bb9d2eebc183ac7c26 (DEFAULT_TABLE table=test-table [id=7a817b6cb18c434e90140872a680a8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:20:16.923350 16818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94109dea54f47bb9d2eebc183ac7c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:20:16.925185 16687 tablet_service.cc:1467] Processing CreateTablet for tablet e94109dea54f47bb9d2eebc183ac7c26 (DEFAULT_TABLE table=test-table [id=7a817b6cb18c434e90140872a680a8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:20:16.926425 16687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94109dea54f47bb9d2eebc183ac7c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:20:16.930467 16899 tablet_bootstrap.cc:492] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Bootstrap starting.
I20260620 15:20:16.934134 16899 tablet_bootstrap.cc:654] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Neither blocks nor log segments found. Creating new log.
I20260620 15:20:16.935464 16899 log.cc:824] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Log is configured to *not* fsync() on all Append() calls
I20260620 15:20:16.938758 16899 tablet_bootstrap.cc:492] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: No bootstrap required, opened a new log
I20260620 15:20:16.939116 16899 ts_tablet_manager.cc:1397] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 15:20:16.947117 16902 tablet_bootstrap.cc:492] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: Bootstrap starting.
I20260620 15:20:16.951319 16901 tablet_bootstrap.cc:492] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: Bootstrap starting.
I20260620 15:20:16.951807 16902 tablet_bootstrap.cc:654] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: Neither blocks nor log segments found. Creating new log.
I20260620 15:20:16.952845 16899 raft_consensus.cc:348] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:16.953114 16902 log.cc:824] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:20:16.953279 16899 raft_consensus.cc:374] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:20:16.953428 16899 raft_consensus.cc:729] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a59e45b9301d4b21b1d72f1f36e4a247, State: Initialized, Role: FOLLOWER
I20260620 15:20:16.954111 16899 consensus_queue.cc:260] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:16.954469 16901 tablet_bootstrap.cc:654] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: Neither blocks nor log segments found. Creating new log.
I20260620 15:20:16.955118 16620 heartbeater.cc:503] Master 127.0.97.126:35917 was elected leader, sending a full tablet report...
I20260620 15:20:16.955801 16899 ts_tablet_manager.cc:1428] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Time spent starting tablet: real 0.017s user 0.008s sys 0.006s
I20260620 15:20:16.956244 16902 tablet_bootstrap.cc:492] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: No bootstrap required, opened a new log
I20260620 15:20:16.956523 16902 ts_tablet_manager.cc:1397] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 15:20:16.956830 16901 log.cc:824] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: Log is configured to *not* fsync() on all Append() calls
I20260620 15:20:16.961004 16901 tablet_bootstrap.cc:492] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: No bootstrap required, opened a new log
I20260620 15:20:16.961280 16901 ts_tablet_manager.cc:1397] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260620 15:20:16.964890 16902 raft_consensus.cc:348] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:16.965237 16902 raft_consensus.cc:374] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:20:16.965356 16902 raft_consensus.cc:729] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efec529f7a1a44d4b5f155724656c9b4, State: Initialized, Role: FOLLOWER
I20260620 15:20:16.965883 16902 consensus_queue.cc:260] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:16.966957 16752 heartbeater.cc:503] Master 127.0.97.126:35917 was elected leader, sending a full tablet report...
I20260620 15:20:16.967559 16902 ts_tablet_manager.cc:1428] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260620 15:20:16.969045 16901 raft_consensus.cc:348] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:16.969367 16901 raft_consensus.cc:374] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:20:16.969473 16901 raft_consensus.cc:729] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c42a3142da7467c92998b03ea3b74df, State: Initialized, Role: FOLLOWER
I20260620 15:20:16.969996 16901 consensus_queue.cc:260] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:16.971164 16883 heartbeater.cc:503] Master 127.0.97.126:35917 was elected leader, sending a full tablet report...
I20260620 15:20:16.971724 16901 ts_tablet_manager.cc:1428] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260620 15:20:17.001286 16906 raft_consensus.cc:482] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:20:17.001581 16906 raft_consensus.cc:504] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:17.003216 16906 leader_election.cc:288] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163), a59e45b9301d4b21b1d72f1f36e4a247 (127.0.97.65:44503)
I20260620 15:20:17.010170 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.003909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:48923 (local address 127.0.97.65:44503)
0620 15:20:17.004134 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:17.004142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:17.004571 (+ 429us) server_negotiation.cc:407] Connection header received
0620 15:20:17.004749 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:17.004755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:17.004832 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:17.004935 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:20:17.005607 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.006387 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.008249 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.009194 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.009344 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:17.009465 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:17.009771 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 15:20:17.009898 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:20:17.010169 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.003489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:59087 (local address 127.0.97.67:39163)
0620 15:20:17.003712 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:17.003721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:17.004300 (+ 579us) server_negotiation.cc:407] Connection header received
0620 15:20:17.004538 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:17.004544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:17.004624 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:17.004721 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:20:17.005361 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.006158 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.008294 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.009171 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.009340 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:17.009474 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:17.009780 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 15:20:17.009908 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:20:17.010385 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.003604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.97.66:48923)
0620 15:20:17.004434 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:17.004457 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:17.004604 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:17.004961 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:17.004964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:17.004988 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:20:17.005471 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:17.005482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.006525 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.006528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:17.008106 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:17.008117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.009433 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.009440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:17.009606 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:17.009630 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:20:17.009736 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 15:20:17.009869 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":58,"thread_start_us":448,"threads_started":1}
I20260620 15:20:17.010422 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.003416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39163 (local address 127.0.97.66:59087)
0620 15:20:17.004163 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:17.004187 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:17.004356 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:17.004762 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:17.004767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:17.004793 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:20:17.005216 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:17.005226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.006302 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.006307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:17.008127 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:17.008142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.009473 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.009480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:17.009628 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:17.009650 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:20:17.009761 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 15:20:17.009892 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":89,"thread_start_us":350,"threads_started":1}
I20260620 15:20:17.011458 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94109dea54f47bb9d2eebc183ac7c26" candidate_uuid: "efec529f7a1a44d4b5f155724656c9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c42a3142da7467c92998b03ea3b74df" is_pre_election: true
I20260620 15:20:17.011448 16575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94109dea54f47bb9d2eebc183ac7c26" candidate_uuid: "efec529f7a1a44d4b5f155724656c9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" is_pre_election: true
I20260620 15:20:17.011934 16575 raft_consensus.cc:2454] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efec529f7a1a44d4b5f155724656c9b4 in term 0.
I20260620 15:20:17.011931 16838 raft_consensus.cc:2454] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efec529f7a1a44d4b5f155724656c9b4 in term 0.
I20260620 15:20:17.012612 16640 leader_election.cc:302] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c42a3142da7467c92998b03ea3b74df, efec529f7a1a44d4b5f155724656c9b4; no voters:
I20260620 15:20:17.013020 16906 raft_consensus.cc:2781] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:20:17.013164 16906 raft_consensus.cc:482] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:20:17.013276 16906 raft_consensus.cc:3037] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:20:17.015573 16906 raft_consensus.cc:504] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:17.016182 16906 leader_election.cc:288] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [CANDIDATE]: Term 1 election: Requested vote from peers 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163), a59e45b9301d4b21b1d72f1f36e4a247 (127.0.97.65:44503)
I20260620 15:20:17.016659 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94109dea54f47bb9d2eebc183ac7c26" candidate_uuid: "efec529f7a1a44d4b5f155724656c9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c42a3142da7467c92998b03ea3b74df"
I20260620 15:20:17.016903 16838 raft_consensus.cc:3037] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:20:17.016889 16575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94109dea54f47bb9d2eebc183ac7c26" candidate_uuid: "efec529f7a1a44d4b5f155724656c9b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a59e45b9301d4b21b1d72f1f36e4a247"
I20260620 15:20:17.017102 16575 raft_consensus.cc:3037] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:20:17.019394 16838 raft_consensus.cc:2454] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efec529f7a1a44d4b5f155724656c9b4 in term 1.
I20260620 15:20:17.019766 16640 leader_election.cc:302] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c42a3142da7467c92998b03ea3b74df, efec529f7a1a44d4b5f155724656c9b4; no voters:
I20260620 15:20:17.019974 16575 raft_consensus.cc:2454] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efec529f7a1a44d4b5f155724656c9b4 in term 1.
I20260620 15:20:17.020036 16906 raft_consensus.cc:2781] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:20:17.020607 16906 raft_consensus.cc:686] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 1 LEADER]: Becoming Leader. State: Replica: efec529f7a1a44d4b5f155724656c9b4, State: Running, Role: LEADER
I20260620 15:20:17.021119 16906 consensus_queue.cc:237] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:17.025558 16434 catalog_manager.cc:5495] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "efec529f7a1a44d4b5f155724656c9b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } health_report { overall_health: UNKNOWN } } }
I20260620 15:20:17.059875 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.054294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.0.97.66:35249)
0620 15:20:17.054475 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:17.054480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:17.054497 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:20:17.054700 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:17.054704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:17.054820 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:17.054945 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:20:17.055376 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.056101 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.057605 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.058024 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.058130 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:17.058645 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:20:17.059223 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:20:17.059337 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:17.059506 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 15:20:17.059577 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:20:17.068596 16838 raft_consensus.cc:1261] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df [term 1 FOLLOWER]: Refusing update from remote peer efec529f7a1a44d4b5f155724656c9b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:20:17.068608 16575 raft_consensus.cc:1261] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Refusing update from remote peer efec529f7a1a44d4b5f155724656c9b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:20:17.069628 16910 consensus_queue.cc:1035] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:20:17.069986 16906 consensus_queue.cc:1035] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:20:17.073657 16884 tablet.cc:2369] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:20:17.093645 16433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:20:17.093983 16433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:20:17.097441 16433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Sending DeleteTablet for 3 replicas of tablet e94109dea54f47bb9d2eebc183ac7c26
I20260620 15:20:17.098740 16818 tablet_service.cc:1514] Processing DeleteTablet for tablet e94109dea54f47bb9d2eebc183ac7c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:20:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40166
I20260620 15:20:17.098989 16687 tablet_service.cc:1514] Processing DeleteTablet for tablet e94109dea54f47bb9d2eebc183ac7c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:20:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49198
I20260620 15:20:17.099478 16555 tablet_service.cc:1514] Processing DeleteTablet for tablet e94109dea54f47bb9d2eebc183ac7c26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:20:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:54730
I20260620 15:20:17.101047 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 16471
I20260620 15:20:17.114840 16923 tablet_replica.cc:330] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: stopping tablet replica
I20260620 15:20:17.115306 16923 raft_consensus.cc:2229] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:20:17.115846 16923 raft_consensus.cc:2258] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:20:17.116662 16925 tablet_replica.cc:330] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: stopping tablet replica
I20260620 15:20:17.117055 16925 raft_consensus.cc:2229] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:20:17.117609 16925 raft_consensus.cc:2258] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:20:17.118639 16924 tablet_replica.cc:330] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: stopping tablet replica
I20260620 15:20:17.119074 16924 raft_consensus.cc:2229] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:20:17.119787 16924 raft_consensus.cc:2258] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:20:17.119921 16923 ts_tablet_manager.cc:1905] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:20:17.121819 16925 ts_tablet_manager.cc:1905] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:20:17.123770 16924 ts_tablet_manager.cc:1905] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:20:17.123415 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.102029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.0.97.65:44503)
0620 15:20:17.102237 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:17.102247 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:17.102272 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:20:17.102432 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:17.102441 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:17.102566 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:17.102705 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:17.104437 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.105863 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.109145 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.110044 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.110201 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:17.114603 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:17.114626 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:17.114654 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:17.114706 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:17.120413 (+ 5707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:17.121184 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:17.121193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:17.121196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:17.121356 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:17.121729 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:17.121734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:17.121740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:17.122233 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:17.122532 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:17.122852 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 15:20:17.123077 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:20:17.124747 16615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:17.128871 16923 ts_tablet_manager.cc:1918] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:20:17.129042 16923 log.cc:1196] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/e94109dea54f47bb9d2eebc183ac7c26
I20260620 15:20:17.129602 16923 ts_tablet_manager.cc:1939] T e94109dea54f47bb9d2eebc183ac7c26 P 5c42a3142da7467c92998b03ea3b74df: Deleting consensus metadata
I20260620 15:20:17.131506 16418 catalog_manager.cc:4841] TS 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163): tablet e94109dea54f47bb9d2eebc183ac7c26 (table test-table [id=7a817b6cb18c434e90140872a680a8d0]) successfully deleted
I20260620 15:20:17.132645 16924 ts_tablet_manager.cc:1918] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:20:17.132862 16924 log.cc:1196] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/e94109dea54f47bb9d2eebc183ac7c26
I20260620 15:20:17.133496 16924 ts_tablet_manager.cc:1939] T e94109dea54f47bb9d2eebc183ac7c26 P efec529f7a1a44d4b5f155724656c9b4: Deleting consensus metadata
I20260620 15:20:17.135375 16418 catalog_manager.cc:4841] TS efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249): tablet e94109dea54f47bb9d2eebc183ac7c26 (table test-table [id=7a817b6cb18c434e90140872a680a8d0]) successfully deleted
I20260620 15:20:17.249281 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 16471
I20260620 15:20:17.267333 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 16624
W20260620 15:20:17.267371 16420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:44503 (error 108)
I20260620 15:20:17.269471 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.268995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:54748)
0620 15:20:17.269209 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:17.269332 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 15:20:17.269790 16420 catalog_manager.cc:4584] TS a59e45b9301d4b21b1d72f1f36e4a247 (127.0.97.65:44503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e94109dea54f47bb9d2eebc183ac7c26: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
I20260620 15:20:17.283883 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.268550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.0.97.66:35249)
0620 15:20:17.268797 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:17.268802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:17.268836 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:20:17.268902 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:17.268908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:17.268990 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:17.269117 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:17.270859 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.271665 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.273564 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.273986 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.274076 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:17.277468 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:17.277480 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:17.277518 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:17.277567 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:17.281116 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:17.281806 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:17.281819 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:17.281822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:17.281966 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:17.282361 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:17.282365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:17.282367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:17.282711 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:17.283049 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:17.283294 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 15:20:17.283537 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 15:20:17.284533 16747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:17.323508 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.322978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:54762)
0620 15:20:17.323185 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:17.323286 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 15:20:17.324540 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.324149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:54778)
0620 15:20:17.324297 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:17.324375 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 15:20:17.384989 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.384472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:54780)
0620 15:20:17.384751 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:17.384840 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 15:20:17.386217 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.385840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:54784)
0620 15:20:17.385999 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:17.386086 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:20:17.398492 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 16624
I20260620 15:20:17.417968 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 16755
I20260620 15:20:17.430742 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.418726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.0.97.67:39163)
0620 15:20:17.418904 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:17.418909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:17.418929 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:20:17.419113 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:17.419117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:17.419179 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:17.419277 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:17.420647 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.421438 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.422730 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:17.423111 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:17.423216 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:17.425684 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:17.425697 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:17.425713 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:17.425743 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:17.428315 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:17.429058 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:17.429064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:17.429067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:17.429191 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:17.429519 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:17.429523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:17.429525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:17.429851 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:17.430055 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:17.430327 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:20:17.430484 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:20:17.431360 16878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:17.466763 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.466316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:54796)
0620 15:20:17.466533 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:17.466627 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 15:20:17.467761 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.467389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:54806)
0620 15:20:17.467586 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:17.467654 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 15:20:17.520046 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 16755
I20260620 15:20:17.537039 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 16401
I20260620 15:20:17.538321 16464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:17.649569 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 16401
I20260620 15:20:17.649607 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:17.648910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44503 (local address 127.0.0.1:54808)
0620 15:20:17.649353 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:17.649453 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:44503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280}
I20260620 15:20:37.670902 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35917
--webserver_interface=127.0.97.126
--webserver_port=41943
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:20:37.988818 16931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:37.989117 16931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:37.989271 16931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:37.996230 16931 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:20:37.996367 16931 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:20:37.996452 16931 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:20:37.996559 16931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:20:37.996662 16931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:37.996763 16931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:37.996848 16931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:20:37.996928 16931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:20:38.007571 16931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=41943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:38.009498 16931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:38.011143 16931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:38.022851 16936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:38.022987 16939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:38.022960 16937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:38.023473 16931 server_base.cc:1034] running on GCE node
I20260620 15:20:38.024277 16931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:38.025614 16931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:38.026844 16931 hybrid_clock.cc:648] HybridClock initialized: now 1781968838026794 us; error 25 us; skew 500 ppm
I20260620 15:20:38.027180 16931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:38 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:38.035244 16931 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:20:38.036813 16931 webserver.cc:460] Webserver started at http://127.0.97.126:41943/ using document root <none> and password file <none>
I20260620 15:20:38.037575 16931 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:38.037739 16931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:38.043406 16931 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 15:20:38.046083 16946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:38.047293 16931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:20:38.047489 16931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "2ca71167a4574b0591d4a58e89ce4ba0"
format_stamp: "Formatted at 2026-06-20 15:19:52 on dist-test-slave-7s64"
server_key: "7f31729cc8c6f72f47d92d9237b6cbeb"
server_key_iv: "ddc442ee3361b1818b4ed91a45d3f18d"
server_key_version: "encryptionkey@0"
I20260620 15:20:38.048390 16931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:38.074565 16931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:38.078532 16931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:38.078827 16931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:38.098013 16931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35917
I20260620 15:20:38.098053 16997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35917 every 8 connection(s)
I20260620 15:20:38.099480 16931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:20:38.101824 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 16931
I20260620 15:20:38.102646 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:44503
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:35917
--webserver_port=40089
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:20:38.106359 16998 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:20:38.109560 16998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Bootstrap starting.
I20260620 15:20:38.129715 16998 log.cc:824] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Log is configured to *not* fsync() on all Append() calls
I20260620 15:20:38.147025 16998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:20:38.147593 16998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Bootstrap complete.
I20260620 15:20:38.157399 16998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:20:38.158288 16998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca71167a4574b0591d4a58e89ce4ba0, State: Initialized, Role: FOLLOWER
I20260620 15:20:38.158926 16998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:20:38.159194 16998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:20:38.159332 16998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:20:38.159524 16998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 15:20:38.163527 16998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:20:38.164033 16998 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ca71167a4574b0591d4a58e89ce4ba0; no voters:
I20260620 15:20:38.164762 16998 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 15:20:38.164916 17002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 15:20:38.166150 17002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [term 3 LEADER]: Becoming Leader. State: Replica: 2ca71167a4574b0591d4a58e89ce4ba0, State: Running, Role: LEADER
I20260620 15:20:38.166749 17002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } }
I20260620 15:20:38.167074 16998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:20:38.169368 17004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca71167a4574b0591d4a58e89ce4ba0. Latest consensus state: current_term: 3 leader_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } } }
I20260620 15:20:38.169757 17004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: This master's current role is: LEADER
I20260620 15:20:38.169485 17003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ca71167a4574b0591d4a58e89ce4ba0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35917 } } }
I20260620 15:20:38.169863 17003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0 [sys.catalog]: This master's current role is: LEADER
I20260620 15:20:38.181457 17008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:20:38.191466 17008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:20:38.193022 17008 catalog_manager.cc:1259] Loaded cluster ID: 8466b4886aee46158e337fb5c47e956a
I20260620 15:20:38.193172 17008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:20:38.197597 17008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:20:38.199172 17008 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Loaded TSK: 0
I20260620 15:20:38.199358 17008 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ca71167a4574b0591d4a58e89ce4ba0: Loaded TSK: 1
F20260620 15:20:38.200456 17008 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f761de209ad google::LogMessage::Fail() at ??:0
@ 0x7f761de2388a google::LogMessage::SendToLog() at ??:0
@ 0x7f761de204de google::LogMessage::Flush() at ??:0
@ 0x7f761de21a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7621b847d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7621b7373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f763384f1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f763384e783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f76338da2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f76338da0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f76339245e9 std::function<>::operator()() at ??:0
@ 0x7f7633852cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f7633850c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f76338d8610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f76338d8439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7620a6c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7620a6f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f7620a6f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f7620a4aeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7623d246db start_thread at ??:0
@ 0x7f761ccba61f clone at ??:0
W20260620 15:20:38.531688 17000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:38.531944 17000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:38.532035 17000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:38.539238 17000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:38.539333 17000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:38.539443 17000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:20:38.548660 17000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:44503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=40089
--tserver_master_addrs=127.0.97.126:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:38.549762 17000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:38.550951 17000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:38.563586 17045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:38.563647 17046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:38.563922 17048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:38.564713 17000 server_base.cc:1034] running on GCE node
I20260620 15:20:38.565367 17000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:38.566228 17000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:38.567451 17000 hybrid_clock.cc:648] HybridClock initialized: now 1781968838567399 us; error 48 us; skew 500 ppm
I20260620 15:20:38.567761 17000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:38 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:38.570783 17000 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:20:38.572207 17000 webserver.cc:460] Webserver started at http://127.0.97.65:40089/ using document root <none> and password file <none>
I20260620 15:20:38.572914 17000 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:38.573067 17000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:38.578105 17000 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:20:38.580976 17055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:38.582252 17000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:20:38.582427 17000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "a59e45b9301d4b21b1d72f1f36e4a247"
format_stamp: "Formatted at 2026-06-20 15:19:53 on dist-test-slave-7s64"
server_key: "e178fdaf7efc66a2f7a7c5c9d4255cb2"
server_key_iv: "c3f94ddb942a5f39253b776e69e1e6e3"
server_key_version: "encryptionkey@0"
I20260620 15:20:38.583406 17000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:38.602687 17000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:38.606433 17000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:38.606698 17000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:38.607604 17000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:20:38.609396 17062 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 15:20:38.614660 17000 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 15:20:38.614816 17000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260620 15:20:38.614945 17000 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 15:20:38.617946 17000 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 15:20:38.618045 17000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260620 15:20:38.618335 17062 tablet_bootstrap.cc:492] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Bootstrap starting.
I20260620 15:20:38.626127 17062 log.cc:824] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Log is configured to *not* fsync() on all Append() calls
I20260620 15:20:38.639946 17062 tablet_bootstrap.cc:492] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:20:38.640862 17062 tablet_bootstrap.cc:492] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Bootstrap complete.
I20260620 15:20:38.641469 17062 ts_tablet_manager.cc:1397] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.016s
I20260620 15:20:38.651607 17062 raft_consensus.cc:348] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:38.653076 17062 raft_consensus.cc:729] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a59e45b9301d4b21b1d72f1f36e4a247, State: Initialized, Role: FOLLOWER
I20260620 15:20:38.653700 17062 consensus_queue.cc:260] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:38.655123 17062 ts_tablet_manager.cc:1428] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260620 15:20:38.659132 17000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:44503
I20260620 15:20:38.659209 17170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:44503 every 8 connection(s)
I20260620 15:20:38.660223 17000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:20:38.664366 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:38.662720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:54645)
0620 15:20:38.663399 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:38.663590 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":286,"threads_started":1}
W20260620 15:20:38.665084 17171 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35917 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:35917: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
I20260620 15:20:38.665349 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 17000
I20260620 15:20:38.665871 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:38.665564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:42513)
0620 15:20:38.665700 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:38.665773 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 15:20:38.666047 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:35249
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:35917
--webserver_port=33059
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:20:38.666700 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:38.666394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:52597)
0620 15:20:38.666527 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:38.666595 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 15:20:38.666941 17171 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:20:38.983217 17174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:38.983456 17174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:38.983608 17174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:38.990675 17174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:38.990767 17174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:38.990962 17174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:20:39.000135 17174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:35249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=33059
--tserver_master_addrs=127.0.97.126:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:39.001336 17174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:39.002614 17174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:39.015631 17179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:39.015672 17182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:39.015517 17180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:39.016590 17174 server_base.cc:1034] running on GCE node
I20260620 15:20:39.017230 17174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:39.018085 17174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:39.019295 17174 hybrid_clock.cc:648] HybridClock initialized: now 1781968839019243 us; error 62 us; skew 500 ppm
I20260620 15:20:39.019590 17174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:39 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:39.022596 17174 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:20:39.023950 17174 webserver.cc:460] Webserver started at http://127.0.97.66:33059/ using document root <none> and password file <none>
I20260620 15:20:39.024670 17174 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:39.024811 17174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:39.029811 17174 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 15:20:39.032531 17189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:39.033802 17174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:20:39.033980 17174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "efec529f7a1a44d4b5f155724656c9b4"
format_stamp: "Formatted at 2026-06-20 15:19:53 on dist-test-slave-7s64"
server_key: "48e73ae2670ca0fbcf32b9a941d3d771"
server_key_iv: "db12dcab2cab4412c008550f60f3378f"
server_key_version: "encryptionkey@0"
I20260620 15:20:39.034930 17174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:39.049813 17174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:39.053274 17174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:39.053521 17174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:39.054378 17174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:20:39.055570 17174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:20:39.055683 17174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:39.055799 17174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:20:39.055883 17174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:39.088800 17174 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:35249
I20260620 15:20:39.088887 17301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:35249 every 8 connection(s)
I20260620 15:20:39.090256 17174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:20:39.094911 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:39.093051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:43715)
0620 15:20:39.093758 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:39.093955 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":272,"threads_started":1}
W20260620 15:20:39.096792 17302 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35917 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:35917: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
I20260620 15:20:39.096925 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 17174
I20260620 15:20:39.097872 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:39163
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:35917
--webserver_port=41521
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:20:39.101261 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:39.100823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:34215)
0620 15:20:39.101004 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:39.101098 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 15:20:39.104100 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:39.103519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:53097)
0620 15:20:39.103877 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:39.103985 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
W20260620 15:20:39.104712 17302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:20:39.423631 17305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:39.423928 17305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:39.424026 17305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:39.431022 17305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:39.431113 17305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:39.431221 17305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:20:39.440646 17305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:39163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=41521
--tserver_master_addrs=127.0.97.126:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:39.441710 17305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:39.442911 17305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:39.455219 17310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:39.455338 17311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:39.455343 17313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:39.456323 17305 server_base.cc:1034] running on GCE node
I20260620 15:20:39.456954 17305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:39.457738 17305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:39.458925 17305 hybrid_clock.cc:648] HybridClock initialized: now 1781968839458876 us; error 38 us; skew 500 ppm
I20260620 15:20:39.459172 17305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:39 dist-test-slave-7s64 krb5kdc[15850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:39.462194 17305 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:20:39.463454 17305 webserver.cc:460] Webserver started at http://127.0.97.67:41521/ using document root <none> and password file <none>
I20260620 15:20:39.464109 17305 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:39.464231 17305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:39.469295 17305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:20:39.471704 17320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:39.472990 17305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:20:39.473158 17305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "5c42a3142da7467c92998b03ea3b74df"
format_stamp: "Formatted at 2026-06-20 15:19:54 on dist-test-slave-7s64"
server_key: "0f326f949aa78cd46c3668931e639e92"
server_key_iv: "e8d2aa033817eab68ce265fb7c1a8e96"
server_key_version: "encryptionkey@0"
I20260620 15:20:39.474073 17305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:39.487869 17305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:39.491770 17305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:39.492022 17305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:39.492928 17305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:20:39.494127 17305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:20:39.494248 17305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:39.494369 17305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:20:39.494464 17305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:39.526785 17305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39163
I20260620 15:20:39.526859 17432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39163 every 8 connection(s)
I20260620 15:20:39.528160 17305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:20:39.532557 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:39.530837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:49171)
0620 15:20:39.531528 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:39.531724 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":276,"threads_started":1}
W20260620 15:20:39.533205 17433 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35917 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:35917: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
I20260620 15:20:39.533955 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:39.533668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:46237)
0620 15:20:39.533794 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:39.533857 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 15:20:39.534718 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:39.534450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:52639)
0620 15:20:39.534559 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:39.534626 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 15:20:39.534952 17433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:20:39.535598 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 17305
I20260620 15:20:39.668154 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:39.667452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:37163)
0620 15:20:39.667901 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:39.667974 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":236,"threads_started":1}
I20260620 15:20:39.999373 17438 raft_consensus.cc:482] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:20:39.999641 17438 raft_consensus.cc:504] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:40.001286 17438 leader_election.cc:288] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163), efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249)
Jun 20 15:20:40 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
Jun 20 15:20:40 dist-test-slave-7s64 krb5kdc[15850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:20:40.023223 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:40.001898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:54825 (local address 127.0.97.67:39163)
0620 15:20:40.003651 (+ 1753us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:40.003667 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:40.003717 (+ 50us) server_negotiation.cc:407] Connection header received
0620 15:20:40.006033 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:40.006041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:40.006488 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:40.006847 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:40.008451 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:40.009652 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:40.011414 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:40.012000 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:40.012236 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:40.016083 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:40.016093 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:40.016111 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:40.016157 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:40.019206 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:40.020164 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:40.020172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:40.020181 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:40.020382 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:40.020938 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:40.020945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:40.020951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:40.021425 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:40.021551 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:40.022418 (+ 867us) server_negotiation.cc:299] Negotiation successful
0620 15:20:40.022780 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329,"thread_start_us":1203,"threads_started":1}
I20260620 15:20:40.023247 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:40.001900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:35249 (local address 127.0.97.65:38569)
0620 15:20:40.003162 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:40.003186 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:40.003323 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:40.007420 (+ 4097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:40.007424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:40.007925 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:20:40.008266 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:40.008284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:40.009834 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:40.009837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:40.011142 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:40.011158 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:40.012147 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:40.012154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:40.012421 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:40.013117 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:20:40.013138 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:20:40.016102 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:20:40.019739 (+ 3637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:40.019742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:40.019745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:40.020200 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:40.020723 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:40.020727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:40.020729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:40.020868 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:40.021652 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:20:40.021655 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:20:40.021942 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 15:20:40.022210 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 15:20:40.022768 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"mutex_wait_us":240,"thread_start_us":921,"threads_started":1}
I20260620 15:20:40.023279 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:40.002178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:38569 (local address 127.0.97.66:35249)
0620 15:20:40.003699 (+ 1521us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:40.003710 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:40.003741 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:20:40.005980 (+ 2239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:40.005992 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:40.006488 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:40.006847 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:40.008459 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:40.009661 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:40.011273 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:40.011961 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:40.012113 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:40.016331 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:40.016338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:40.016349 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:40.016387 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:40.019584 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:40.020337 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:40.020344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:40.020349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:40.020599 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:40.020994 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:40.020998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:40.021000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:40.021511 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:40.021692 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:40.022439 (+ 747us) server_negotiation.cc:299] Negotiation successful
0620 15:20:40.022797 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":1228,"threads_started":1}
I20260620 15:20:40.023379 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:40.001492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39163 (local address 127.0.97.65:54825)
0620 15:20:40.001685 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:40.001719 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:40.002268 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:40.006966 (+ 4698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:40.006975 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:40.007581 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:20:40.008266 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:40.008283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:40.009833 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:40.009837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:40.011273 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:40.011287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:40.012214 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:40.012221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:40.012413 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:40.013276 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:20:40.013293 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:20:40.015797 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:20:40.019360 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:40.019372 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:40.019388 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:40.020027 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:40.020553 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:40.020559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:40.020562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:40.020818 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:40.021621 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:20:40.021630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:20:40.021961 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 15:20:40.022211 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 15:20:40.022927 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":261,"spinlock_wait_cycles":37376}
W20260620 15:20:40.026649 17056 leader_election.cc:341] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249): Not found: Tablet not found: e94109dea54f47bb9d2eebc183ac7c26
W20260620 15:20:40.026993 17056 leader_election.cc:341] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163): Not found: Tablet not found: e94109dea54f47bb9d2eebc183ac7c26
I20260620 15:20:40.027139 17056 leader_election.cc:302] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a59e45b9301d4b21b1d72f1f36e4a247; no voters: 5c42a3142da7467c92998b03ea3b74df, efec529f7a1a44d4b5f155724656c9b4
I20260620 15:20:40.027545 17438 raft_consensus.cc:2726] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:20:40.106016 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:40.105157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:53701)
0620 15:20:40.105625 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:40.105774 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":230,"threads_started":1}
I20260620 15:20:40.536176 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:40.535478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:40517)
0620 15:20:40.535869 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:40.535970 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":191,"threads_started":1}
I20260620 15:20:40.669754 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:40.669026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:56055)
0620 15:20:40.669510 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:40.669594 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":238,"threads_started":1}
I20260620 15:20:41.107692 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:41.106918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:46491)
0620 15:20:41.107426 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:41.107520 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260620 15:20:41.537905 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:41.537093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:52557)
0620 15:20:41.537601 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:41.537702 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":289,"threads_started":1}
I20260620 15:20:41.671465 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:41.670614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:33163)
0620 15:20:41.671117 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:41.671241 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":255,"threads_started":1}
I20260620 15:20:41.978518 17448 raft_consensus.cc:482] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:20:41.978749 17448 raft_consensus.cc:504] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:41.979374 17448 leader_election.cc:288] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163), efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249)
W20260620 15:20:41.980466 17056 leader_election.cc:341] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163): Not found: Tablet not found: e94109dea54f47bb9d2eebc183ac7c26
W20260620 15:20:41.980770 17056 leader_election.cc:341] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249): Not found: Tablet not found: e94109dea54f47bb9d2eebc183ac7c26
I20260620 15:20:41.980855 17056 leader_election.cc:302] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a59e45b9301d4b21b1d72f1f36e4a247; no voters: 5c42a3142da7467c92998b03ea3b74df, efec529f7a1a44d4b5f155724656c9b4
I20260620 15:20:41.981127 17448 raft_consensus.cc:2726] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:20:42.109310 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:42.108582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:33027)
0620 15:20:42.109048 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:42.109135 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
I20260620 15:20:42.539534 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:42.538716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:38879)
0620 15:20:42.539267 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:42.539382 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":225,"threads_started":1}
I20260620 15:20:42.672976 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:42.672330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:47885)
0620 15:20:42.672753 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:42.672827 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":245,"threads_started":1}
I20260620 15:20:43.110836 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:43.110189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:41441)
0620 15:20:43.110578 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:43.110653 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":193,"threads_started":1}
I20260620 15:20:43.541146 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:43.540346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:54461)
0620 15:20:43.540892 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:43.540960 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":315,"threads_started":1}
I20260620 15:20:43.674542 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:43.673847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:56323)
0620 15:20:43.674288 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:43.674375 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":201,"threads_started":1}
I20260620 15:20:44.112541 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:44.111796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:51583)
0620 15:20:44.112237 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:44.112326 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":236,"threads_started":1}
I20260620 15:20:44.428682 17456 raft_consensus.cc:482] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:20:44.428891 17456 raft_consensus.cc:504] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:44.429539 17456 leader_election.cc:288] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163), efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249)
W20260620 15:20:44.430503 17056 leader_election.cc:341] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163): Not found: Tablet not found: e94109dea54f47bb9d2eebc183ac7c26
W20260620 15:20:44.430711 17056 leader_election.cc:341] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249): Not found: Tablet not found: e94109dea54f47bb9d2eebc183ac7c26
I20260620 15:20:44.430847 17056 leader_election.cc:302] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a59e45b9301d4b21b1d72f1f36e4a247; no voters: 5c42a3142da7467c92998b03ea3b74df, efec529f7a1a44d4b5f155724656c9b4
I20260620 15:20:44.431094 17456 raft_consensus.cc:2726] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:20:44.542703 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:44.542053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:34509)
0620 15:20:44.542477 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:44.542554 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":224,"threads_started":1}
I20260620 15:20:44.676232 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:44.675444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:35377)
0620 15:20:44.675931 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:44.676023 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":249,"threads_started":1}
I20260620 15:20:45.114210 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:45.113506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:55287)
0620 15:20:45.113965 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:45.114042 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":258,"threads_started":1}
I20260620 15:20:45.544212 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:45.543552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:40723)
0620 15:20:45.543976 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:45.544048 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260620 15:20:45.677842 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:45.677108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:34089)
0620 15:20:45.677558 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:45.677660 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":210,"threads_started":1}
I20260620 15:20:46.101930 17462 raft_consensus.cc:482] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:20:46.102162 17462 raft_consensus.cc:504] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:46.102711 17462 leader_election.cc:288] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163), efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249)
W20260620 15:20:46.103654 17056 leader_election.cc:341] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163): Not found: Tablet not found: e94109dea54f47bb9d2eebc183ac7c26
W20260620 15:20:46.103828 17056 leader_election.cc:341] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249): Not found: Tablet not found: e94109dea54f47bb9d2eebc183ac7c26
I20260620 15:20:46.103897 17056 leader_election.cc:302] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a59e45b9301d4b21b1d72f1f36e4a247; no voters: 5c42a3142da7467c92998b03ea3b74df, efec529f7a1a44d4b5f155724656c9b4
I20260620 15:20:46.104096 17462 raft_consensus.cc:2726] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:20:46.116052 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:46.115185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:59315)
0620 15:20:46.115788 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:46.115880 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":269,"threads_started":1}
I20260620 15:20:46.545680 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:46.544938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:43837)
0620 15:20:46.545417 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:46.545488 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":225,"threads_started":1}
I20260620 15:20:46.679276 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:46.678624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:58555)
0620 15:20:46.679050 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:46.679125 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":219,"threads_started":1}
I20260620 15:20:47.117697 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:47.117005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:49701)
0620 15:20:47.117407 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:47.117485 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":213,"threads_started":1}
I20260620 15:20:47.547166 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:47.546528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:50007)
0620 15:20:47.546934 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:47.547006 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":226,"threads_started":1}
I20260620 15:20:47.680799 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:47.680061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:45625)
0620 15:20:47.680569 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:47.680644 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":251,"threads_started":1}
I20260620 15:20:48.119339 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:48.118630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:58075)
0620 15:20:48.119092 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:48.119167 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
I20260620 15:20:48.548893 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:48.548073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:50135)
0620 15:20:48.548619 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:48.548723 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":248,"threads_started":1}
I20260620 15:20:48.682344 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:48.681663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:54495)
0620 15:20:48.682102 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:48.682188 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
I20260620 15:20:49.121222 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:49.120387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:55057)
0620 15:20:49.120867 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:49.120967 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":272,"threads_started":1}
I20260620 15:20:49.550433 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:49.549797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:37519)
0620 15:20:49.550195 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:49.550271 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":217,"threads_started":1}
I20260620 15:20:49.683851 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:49.683187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:51557)
0620 15:20:49.683599 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:49.683682 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
I20260620 15:20:50.122831 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:50.122122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:55073)
0620 15:20:50.122589 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:50.122679 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260620 15:20:50.551987 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:50.551357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:58085)
0620 15:20:50.551780 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:50.551851 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":212,"threads_started":1}
I20260620 15:20:50.685524 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:50.684832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:57551)
0620 15:20:50.685284 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:50.685369 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260620 15:20:51.124424 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:51.123719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:48969)
0620 15:20:51.124175 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:51.124253 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":236,"threads_started":1}
I20260620 15:20:51.553540 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:51.552892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:42593)
0620 15:20:51.553296 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:51.553368 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":209,"threads_started":1}
I20260620 15:20:51.687069 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:51.686345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:38219)
0620 15:20:51.686823 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:51.686907 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":252,"threads_started":1}
I20260620 15:20:52.126150 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:52.125379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:49251)
0620 15:20:52.125869 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:52.125968 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":282,"threads_started":1}
I20260620 15:20:52.431440 17482 raft_consensus.cc:482] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:20:52.431612 17482 raft_consensus.cc:504] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c42a3142da7467c92998b03ea3b74df" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39163 } } peers { permanent_uuid: "efec529f7a1a44d4b5f155724656c9b4" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35249 } } peers { permanent_uuid: "a59e45b9301d4b21b1d72f1f36e4a247" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44503 } }
I20260620 15:20:52.432245 17482 leader_election.cc:288] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163), efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249)
W20260620 15:20:52.433202 17056 leader_election.cc:341] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c42a3142da7467c92998b03ea3b74df (127.0.97.67:39163): Not found: Tablet not found: e94109dea54f47bb9d2eebc183ac7c26
W20260620 15:20:52.433427 17056 leader_election.cc:341] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efec529f7a1a44d4b5f155724656c9b4 (127.0.97.66:35249): Not found: Tablet not found: e94109dea54f47bb9d2eebc183ac7c26
I20260620 15:20:52.433578 17056 leader_election.cc:302] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a59e45b9301d4b21b1d72f1f36e4a247; no voters: 5c42a3142da7467c92998b03ea3b74df, efec529f7a1a44d4b5f155724656c9b4
I20260620 15:20:52.433827 17482 raft_consensus.cc:2726] T e94109dea54f47bb9d2eebc183ac7c26 P a59e45b9301d4b21b1d72f1f36e4a247 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:20:52.555049 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:52.554374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:44399)
0620 15:20:52.554826 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:52.554910 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":222,"threads_started":1}
I20260620 15:20:52.688862 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:52.687954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:59729)
0620 15:20:52.688412 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:52.688568 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260620 15:20:53.127750 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:53.127045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:34719)
0620 15:20:53.127480 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:53.127555 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":224,"threads_started":1}
I20260620 15:20:53.556813 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:53.555992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:53123)
0620 15:20:53.556448 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:53.556586 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":251,"threads_started":1}
I20260620 15:20:53.690855 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:53.690136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.65:60343)
0620 15:20:53.690601 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:53.690695 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":266,"threads_started":1}
I20260620 15:20:54.129853 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:54.129118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.66:55821)
0620 15:20:54.129551 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:54.129671 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":231,"threads_started":1}
I20260620 15:20:54.546634 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 17000
I20260620 15:20:54.558547 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:54.557763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35917 (local address 127.0.97.67:42931)
0620 15:20:54.558272 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:54.558378 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":293,"threads_started":1}
I20260620 15:20:54.567126 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:54.547576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.0.97.65:44503)
0620 15:20:54.548220 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:54.548233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:54.548269 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:20:54.548355 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:54.548365 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:54.548679 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:54.548860 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:54.550579 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:54.551612 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:54.552946 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:54.553513 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:54.553665 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:54.561507 (+ 7842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:54.561531 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:54.561544 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:54.561579 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:54.564333 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:54.565066 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:54.565073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:54.565076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:54.565213 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:54.565600 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:54.565605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:54.565607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:54.565948 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:54.566267 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:54.566551 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:20:54.566827 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":282,"threads_started":1}
I20260620 15:20:54.568400 17166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:54.656778 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 17000
I20260620 15:20:54.674124 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 17174
I20260620 15:20:54.687649 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:54.674909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.0.97.66:35249)
0620 15:20:54.675382 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:54.675386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:54.675400 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:20:54.675456 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:54.675460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:54.675518 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:54.675632 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:54.677062 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:54.677949 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:54.679212 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:54.679655 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:54.679789 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:54.682412 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:54.682417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:54.682420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:54.682452 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:54.685056 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:54.685727 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:54.685734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:54.685737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:54.685872 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:54.686254 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:54.686260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:54.686264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:54.686509 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:54.686787 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:54.687067 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 15:20:54.687281 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":235,"threads_started":1}
I20260620 15:20:54.688637 17297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:54.773540 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 17174
I20260620 15:20:54.790992 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 17305
I20260620 15:20:54.804239 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:54.791756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.0.97.67:39163)
0620 15:20:54.792340 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:54.792345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:54.792361 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:20:54.792413 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:54.792416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:54.792532 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:54.792665 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:54.793968 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:54.794809 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:54.795984 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:54.796391 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:54.796647 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:54.799057 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:54.799062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:54.799065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:54.799097 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:54.801894 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:54.802476 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:54.802480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:54.802482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:54.802605 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:54.802977 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:54.802981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:54.802983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:54.803236 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:54.803441 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:54.803674 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 15:20:54.803827 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":320,"threads_started":1}
I20260620 15:20:54.804986 17428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:54.887235 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 17305
2026-06-20T15:20:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (63965 ms)
[----------] 30 tests from SecurityITest (288424 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:20:54 dist-test-slave-7s64 krb5kdc[17498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:20:54 dist-test-slave-7s64 krb5kdc[17498](info): set up 2 sockets
Jun 20 15:20:54 dist-test-slave-7s64 krb5kdc[17498](info): commencing operation
krb5kdc: starting...
W20260620 15:20:55.984592 389 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:20:56 dist-test-slave-7s64 krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:20:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:20:56.205076 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:41515
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:41515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:20:56.522956 17511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:56.523190 17511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:56.523335 17511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:56.530092 17511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:20:56.530177 17511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:56.530252 17511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:56.530333 17511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:20:56.530390 17511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:20:56.539594 17511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:41515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:41515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:56.540861 17511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:56.542100 17511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:56.553671 17516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:56.553722 17517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:56.553748 17519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:56.554219 17511 server_base.cc:1034] running on GCE node
I20260620 15:20:56.554945 17511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:56.556187 17511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:56.557374 17511 hybrid_clock.cc:648] HybridClock initialized: now 1781968856557296 us; error 67 us; skew 500 ppm
I20260620 15:20:56.557647 17511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:56 dist-test-slave-7s64 krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:56.560719 17511 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:20:56.562049 17511 webserver.cc:460] Webserver started at http://127.0.97.126:42805/ using document root <none> and password file <none>
I20260620 15:20:56.562640 17511 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:56.562742 17511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:56.562978 17511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:20:56.565691 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "212284d256404d98a592dbdeac07a669"
format_stamp: "Formatted at 2026-06-20 15:20:56 on dist-test-slave-7s64"
server_key: "fe9a0f7d49c045ec20ba94f879135b5b"
server_key_iv: "026dc7f93873bc349584f7c41f1dafea"
server_key_version: "encryptionkey@0"
I20260620 15:20:56.566345 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "212284d256404d98a592dbdeac07a669"
format_stamp: "Formatted at 2026-06-20 15:20:56 on dist-test-slave-7s64"
server_key: "fe9a0f7d49c045ec20ba94f879135b5b"
server_key_iv: "026dc7f93873bc349584f7c41f1dafea"
server_key_version: "encryptionkey@0"
I20260620 15:20:56.571043 17511 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:20:56.574193 17526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:56.575363 17511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:20:56.575528 17511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "212284d256404d98a592dbdeac07a669"
format_stamp: "Formatted at 2026-06-20 15:20:56 on dist-test-slave-7s64"
server_key: "fe9a0f7d49c045ec20ba94f879135b5b"
server_key_iv: "026dc7f93873bc349584f7c41f1dafea"
server_key_version: "encryptionkey@0"
I20260620 15:20:56.575714 17511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:56.592969 17511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:56.596923 17511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:56.597200 17511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:56.614917 17511 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:41515
I20260620 15:20:56.614952 17577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:41515 every 8 connection(s)
I20260620 15:20:56.616194 17511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:20:56.619688 17578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:20:56.623488 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 17511
I20260620 15:20:56.623730 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:20:56.624538 389 external_mini_cluster.cc:1428] Setting key d4b0255763ea6fc60a90bed253397171
I20260620 15:20:56.633437 17578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669: Bootstrap starting.
I20260620 15:20:56.637203 17578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669: Neither blocks nor log segments found. Creating new log.
I20260620 15:20:56.638531 17578 log.cc:824] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:20:56 dist-test-slave-7s64 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:20:56.641611 17578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669: No bootstrap required, opened a new log
I20260620 15:20:56.647538 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:56.626656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45516 (local address 127.0.97.126:41515)
0620 15:20:56.627375 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:56.627386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:56.627478 (+ 92us) server_negotiation.cc:407] Connection header received
0620 15:20:56.629144 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:56.629167 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:56.629554 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:56.629867 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:56.631732 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:56.633127 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:56.634910 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:56.635583 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:56.635733 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:56.640383 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:56.640390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:56.640404 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:56.640436 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:56.643640 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:56.644331 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:56.644339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:56.644353 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:56.644580 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:56.644977 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:56.644982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:56.644985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:56.645484 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:56.645811 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:56.646208 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 15:20:56.646602 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":347,"threads_started":1}
I20260620 15:20:56.649893 17578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212284d256404d98a592dbdeac07a669" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41515 } }
I20260620 15:20:56.650358 17578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:20:56.650501 17578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212284d256404d98a592dbdeac07a669, State: Initialized, Role: FOLLOWER
I20260620 15:20:56.651082 17578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212284d256404d98a592dbdeac07a669" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41515 } }
I20260620 15:20:56.651324 17578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:20:56.651485 17578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:20:56.651660 17578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:20:56.653702 17578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212284d256404d98a592dbdeac07a669" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41515 } }
I20260620 15:20:56.654155 17578 leader_election.cc:302] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 212284d256404d98a592dbdeac07a669; no voters:
I20260620 15:20:56.654731 17578 leader_election.cc:288] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:20:56.654897 17583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:20:56.655645 17583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [term 1 LEADER]: Becoming Leader. State: Replica: 212284d256404d98a592dbdeac07a669, State: Running, Role: LEADER
I20260620 15:20:56.656286 17583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212284d256404d98a592dbdeac07a669" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41515 } }
I20260620 15:20:56.656903 17578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:20:56.659353 17584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "212284d256404d98a592dbdeac07a669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212284d256404d98a592dbdeac07a669" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41515 } } }
I20260620 15:20:56.659896 17584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [sys.catalog]: This master's current role is: LEADER
I20260620 15:20:56.659327 17585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212284d256404d98a592dbdeac07a669. Latest consensus state: current_term: 1 leader_uuid: "212284d256404d98a592dbdeac07a669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212284d256404d98a592dbdeac07a669" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41515 } } }
I20260620 15:20:56.660290 17585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669 [sys.catalog]: This master's current role is: LEADER
I20260620 15:20:56.665989 17589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:20:56.670413 17589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:20:56.677656 17589 catalog_manager.cc:1347] Generated new cluster ID: 2fca4d1b7b1b40b08f9a9112c20d4dbf
I20260620 15:20:56.677757 17589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:20:56.712234 17589 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:20:56.714123 17589 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:20:56.729410 17589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669: Generated new TSK 0
I20260620 15:20:56.730376 17589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:20:56.789896 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41515
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:20:57.108069 17604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:57.108388 17604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:57.108578 17604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:57.115732 17604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:57.115866 17604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:57.116047 17604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:20:57.125411 17604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:57.126713 17604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:57.128033 17604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:57.141999 17610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:57.142119 17609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:57.142599 17612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:57.142812 17604 server_base.cc:1034] running on GCE node
I20260620 15:20:57.143397 17604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:57.144279 17604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:57.145512 17604 hybrid_clock.cc:648] HybridClock initialized: now 1781968857145445 us; error 72 us; skew 500 ppm
I20260620 15:20:57.145867 17604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:57 dist-test-slave-7s64 krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:57.149174 17604 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:20:57.150485 17604 webserver.cc:460] Webserver started at http://127.0.97.65:44403/ using document root <none> and password file <none>
I20260620 15:20:57.151096 17604 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:57.151229 17604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:57.151475 17604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:20:57.154210 17604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "0b1c496fa7054e48976412e38b0de735"
format_stamp: "Formatted at 2026-06-20 15:20:57 on dist-test-slave-7s64"
server_key: "349fb8d6d81fd4afe8d1be19f8fe3dc5"
server_key_iv: "d14b319766f6c4c146854acc95652efb"
server_key_version: "encryptionkey@0"
I20260620 15:20:57.154901 17604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "0b1c496fa7054e48976412e38b0de735"
format_stamp: "Formatted at 2026-06-20 15:20:57 on dist-test-slave-7s64"
server_key: "349fb8d6d81fd4afe8d1be19f8fe3dc5"
server_key_iv: "d14b319766f6c4c146854acc95652efb"
server_key_version: "encryptionkey@0"
I20260620 15:20:57.159649 17604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 15:20:57.163045 17619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:57.164368 17604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:20:57.164574 17604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "0b1c496fa7054e48976412e38b0de735"
format_stamp: "Formatted at 2026-06-20 15:20:57 on dist-test-slave-7s64"
server_key: "349fb8d6d81fd4afe8d1be19f8fe3dc5"
server_key_iv: "d14b319766f6c4c146854acc95652efb"
server_key_version: "encryptionkey@0"
I20260620 15:20:57.164793 17604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:57.177212 17604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:57.181305 17604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:57.181566 17604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:57.182456 17604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:20:57.183688 17604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:20:57.183758 17604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:57.183905 17604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:20:57.183964 17604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:57.217067 17604 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:43321
I20260620 15:20:57.217154 17731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:43321 every 8 connection(s)
I20260620 15:20:57.218286 17604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:20:57.220201 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 17604
I20260620 15:20:57.220436 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:20:57.221297 389 external_mini_cluster.cc:1428] Setting key 1eb592fcf235fe85c2fb9433d2d417ef
Jun 20 15:20:57 dist-test-slave-7s64 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:20:57.239427 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:57.221066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:55515 (local address 127.0.97.126:41515)
0620 15:20:57.221522 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:57.221527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:57.222697 (+ 1170us) server_negotiation.cc:407] Connection header received
0620 15:20:57.224876 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:57.224882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:57.224971 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:57.225089 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:57.226846 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:57.227878 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:57.229866 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:57.230265 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:57.230400 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:57.233903 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:57.233908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:57.233911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:57.233940 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:57.236472 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:57.237211 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:57.237217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:57.237220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:57.237345 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:57.237764 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:57.237769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:57.237772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:57.238030 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:57.238194 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:57.238828 (+ 634us) server_negotiation.cc:299] Negotiation successful
0620 15:20:57.239068 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":249,"threads_started":1}
I20260620 15:20:57.240272 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:57.221668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:41515 (local address 127.0.97.65:55515)
0620 15:20:57.222526 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:57.222571 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:57.224610 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:57.225321 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:57.225329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:57.225951 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:20:57.226666 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:57.226690 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:57.228059 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:57.228066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:57.229726 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:57.229738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:57.230479 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:57.230487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:57.230707 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:57.231399 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:20:57.231434 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:20:57.233693 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:20:57.236676 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:57.236684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:57.236698 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:57.237084 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:57.237453 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:57.237459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:57.237462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:57.237647 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:57.238160 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:20:57.238167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:20:57.238482 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 15:20:57.238730 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 15:20:57.239208 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":380,"threads_started":1}
I20260620 15:20:57.241982 17732 heartbeater.cc:347] Connected to a master server at 127.0.97.126:41515
I20260620 15:20:57.242295 17732 heartbeater.cc:464] Registering TS with master...
I20260620 15:20:57.242977 17732 heartbeater.cc:511] Master 127.0.97.126:41515 requested a full tablet report, sending...
I20260620 15:20:57.245479 17543 ts_manager.cc:194] Registered new tserver with Master: 0b1c496fa7054e48976412e38b0de735 (127.0.97.65:43321)
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260620 15:20:57.247152 17543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:55515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:20:57.273670 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41515
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:20:57.591503 17738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:57.591895 17738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:57.592115 17738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:57.599481 17738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:57.599571 17738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:57.599763 17738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:20:57.609198 17738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:57.610383 17738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:57.611651 17738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:57.624629 17746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:57.624652 17744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:57.629650 17743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:57.631104 17738 server_base.cc:1034] running on GCE node
I20260620 15:20:57.631610 17738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:57.632473 17738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:57.633752 17738 hybrid_clock.cc:648] HybridClock initialized: now 1781968857633706 us; error 42 us; skew 500 ppm
I20260620 15:20:57.634070 17738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:57 dist-test-slave-7s64 krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:57.637135 17738 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:20:57.638471 17738 webserver.cc:460] Webserver started at http://127.0.97.66:37501/ using document root <none> and password file <none>
I20260620 15:20:57.639138 17738 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:57.639278 17738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:57.639567 17738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:20:57.642355 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "d3b68df98d004936b1b4ca279e1fced5"
format_stamp: "Formatted at 2026-06-20 15:20:57 on dist-test-slave-7s64"
server_key: "221481dcce938ea9ab32d0a959eb096b"
server_key_iv: "5cfbad0de91476207069f9bb4420d219"
server_key_version: "encryptionkey@0"
I20260620 15:20:57.643082 17738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "d3b68df98d004936b1b4ca279e1fced5"
format_stamp: "Formatted at 2026-06-20 15:20:57 on dist-test-slave-7s64"
server_key: "221481dcce938ea9ab32d0a959eb096b"
server_key_iv: "5cfbad0de91476207069f9bb4420d219"
server_key_version: "encryptionkey@0"
I20260620 15:20:57.647712 17738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 15:20:57.650919 17753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:57.652153 17738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:20:57.652316 17738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "d3b68df98d004936b1b4ca279e1fced5"
format_stamp: "Formatted at 2026-06-20 15:20:57 on dist-test-slave-7s64"
server_key: "221481dcce938ea9ab32d0a959eb096b"
server_key_iv: "5cfbad0de91476207069f9bb4420d219"
server_key_version: "encryptionkey@0"
I20260620 15:20:57.652491 17738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:57.678216 17738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:57.682286 17738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:57.682554 17738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:57.683427 17738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:20:57.684706 17738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:20:57.684810 17738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:57.684954 17738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:20:57.685027 17738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:57.718168 17738 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37575
I20260620 15:20:57.718242 17865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37575 every 8 connection(s)
I20260620 15:20:57.719308 17738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:20:57.723826 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 17738
I20260620 15:20:57.724020 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:20:57.724613 389 external_mini_cluster.cc:1428] Setting key 083eabf6e4b9a4838118fa8373c12341
Jun 20 15:20:57 dist-test-slave-7s64 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:20:57.741079 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:57.721771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:43627 (local address 127.0.97.126:41515)
0620 15:20:57.721948 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:57.721953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:57.722966 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 15:20:57.725478 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:57.725484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:57.725572 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:57.725695 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:57.727222 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:57.727962 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:57.730393 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:57.730775 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:57.730873 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:57.734958 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:57.734962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:57.734965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:57.734995 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:57.738074 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:57.738782 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:57.738787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:57.738791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:57.738924 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:57.739350 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:57.739355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:57.739358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:57.739613 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:57.739777 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:57.740460 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 15:20:57.740725 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:20:57.741958 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:57.722139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:41515 (local address 127.0.97.66:43627)
0620 15:20:57.722797 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:57.722840 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:57.725210 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:57.725902 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:57.725910 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:57.726400 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:20:57.727033 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:57.727056 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:57.728128 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:57.728135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:57.730239 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:57.730261 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:57.730982 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:57.730992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:57.731325 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:57.732037 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:20:57.732071 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:20:57.734748 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:20:57.738222 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:57.738233 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:57.738247 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:57.738658 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:57.739043 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:57.739049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:57.739052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:57.739232 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:57.739748 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:20:57.739755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:20:57.740068 (+ 313us) client_negotiation.cc:769] Sending connection context
0620 15:20:57.740352 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 15:20:57.740861 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":265,"threads_started":1}
I20260620 15:20:57.743368 17866 heartbeater.cc:347] Connected to a master server at 127.0.97.126:41515
I20260620 15:20:57.743692 17866 heartbeater.cc:464] Registering TS with master...
I20260620 15:20:57.744576 17866 heartbeater.cc:511] Master 127.0.97.126:41515 requested a full tablet report, sending...
I20260620 15:20:57.746090 17543 ts_manager.cc:194] Registered new tserver with Master: d3b68df98d004936b1b4ca279e1fced5 (127.0.97.66:37575)
I20260620 15:20:57.746893 17543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:43627
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:20:57.779860 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41515
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:20:58.098014 17871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:58.098248 17871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:58.098333 17871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:58.105384 17871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:58.105480 17871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:58.105588 17871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:20:58.114702 17871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:58.115756 17871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:58.116976 17871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:58.129832 17876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:58.129779 17877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:58.129875 17879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:58.131255 17871 server_base.cc:1034] running on GCE node
I20260620 15:20:58.131803 17871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:58.132781 17871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:58.133994 17871 hybrid_clock.cc:648] HybridClock initialized: now 1781968858133942 us; error 50 us; skew 500 ppm
I20260620 15:20:58.134289 17871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:20:58 dist-test-slave-7s64 krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:20:58.137293 17871 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:20:58.138708 17871 webserver.cc:460] Webserver started at http://127.0.97.67:36977/ using document root <none> and password file <none>
I20260620 15:20:58.139350 17871 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:58.139492 17871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:58.139782 17871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:20:58.142493 17871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "15be588632c443a18ab878fc9b295e4b"
format_stamp: "Formatted at 2026-06-20 15:20:58 on dist-test-slave-7s64"
server_key: "c5d92f92dbb676856abfc0aae98e3cac"
server_key_iv: "d9fe55ea7c2f90db20f934c973b8008e"
server_key_version: "encryptionkey@0"
I20260620 15:20:58.143251 17871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "15be588632c443a18ab878fc9b295e4b"
format_stamp: "Formatted at 2026-06-20 15:20:58 on dist-test-slave-7s64"
server_key: "c5d92f92dbb676856abfc0aae98e3cac"
server_key_iv: "d9fe55ea7c2f90db20f934c973b8008e"
server_key_version: "encryptionkey@0"
I20260620 15:20:58.147877 17871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:20:58.151189 17886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:58.152446 17871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:20:58.152647 17871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "15be588632c443a18ab878fc9b295e4b"
format_stamp: "Formatted at 2026-06-20 15:20:58 on dist-test-slave-7s64"
server_key: "c5d92f92dbb676856abfc0aae98e3cac"
server_key_iv: "d9fe55ea7c2f90db20f934c973b8008e"
server_key_version: "encryptionkey@0"
I20260620 15:20:58.152825 17871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:58.177523 17871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:58.181222 17871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:58.181496 17871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:58.182408 17871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:20:58.183624 17871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:20:58.183719 17871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:58.183862 17871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:20:58.183933 17871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:58.217257 17871 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:38935
I20260620 15:20:58.217298 17998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:38935 every 8 connection(s)
I20260620 15:20:58.218747 17871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:20:58.219341 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 17871
I20260620 15:20:58.219547 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:20:58.220165 389 external_mini_cluster.cc:1428] Setting key eff305b8f19c5caf4095ea80c3a41686
Jun 20 15:20:58 dist-test-slave-7s64 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:20:58.241529 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.223386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59919 (local address 127.0.97.126:41515)
0620 15:20:58.223602 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.223611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.224801 (+ 1190us) server_negotiation.cc:407] Connection header received
0620 15:20:58.226996 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.227002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.227102 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.227218 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:58.228713 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.229491 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.231063 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.231553 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.231699 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.235433 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:58.235437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:58.235440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:58.235472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:58.238981 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:58.239623 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:58.239627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:58.239630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:58.239759 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:58.240121 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:58.240124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:58.240126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:58.240400 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:58.240560 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.241114 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.241283 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:20:58.242427 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.222818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:41515 (local address 127.0.97.67:59919)
0620 15:20:58.224646 (+ 1828us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:58.224696 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:58.226732 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:58.227441 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:58.227449 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:58.227955 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:20:58.228483 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.228550 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.229660 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.229666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.230899 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.230912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.231781 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.231791 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.232054 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.232780 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:20:58.232802 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:20:58.235237 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:20:58.239120 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:58.239126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:58.239136 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:58.239516 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:58.239847 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:20:58.239850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:20:58.239852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:20:58.240021 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:20:58.240548 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:20:58.240553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:20:58.240815 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 15:20:58.241020 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 15:20:58.241362 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1522,"thread_start_us":1289,"threads_started":1}
I20260620 15:20:58.243816 17999 heartbeater.cc:347] Connected to a master server at 127.0.97.126:41515
I20260620 15:20:58.244076 17999 heartbeater.cc:464] Registering TS with master...
I20260620 15:20:58.244777 17999 heartbeater.cc:511] Master 127.0.97.126:41515 requested a full tablet report, sending...
I20260620 15:20:58.246093 17543 ts_manager.cc:194] Registered new tserver with Master: 15be588632c443a18ab878fc9b295e4b (127.0.97.67:38935)
I20260620 15:20:58.247149 17543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:59919
I20260620 15:20:58.248370 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:20:58.249925 17732 heartbeater.cc:503] Master 127.0.97.126:41515 was elected leader, sending a full tablet report...
I20260620 15:20:58.264995 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.252812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.0.97.126:41515)
0620 15:20:58.253016 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.253024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.253350 (+ 326us) server_negotiation.cc:407] Connection header received
0620 15:20:58.253484 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.253488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.253549 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.253630 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:58.255061 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.255964 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.257462 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.257852 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.257946 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.260139 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:58.260150 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:58.260153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:58.260179 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:58.262557 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:58.263206 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:58.263210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:58.263212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:58.263335 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:58.263672 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:58.263676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:58.263678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:58.264012 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:58.264201 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.264471 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.264680 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:20:58.268795 17543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:20:58.275403 17543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:20:58.296782 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.286909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.0.97.66:37575)
0620 15:20:58.287646 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.287655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.287764 (+ 109us) server_negotiation.cc:407] Connection header received
0620 15:20:58.287985 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.287996 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.288261 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.288461 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:20:58.289414 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.291467 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.294214 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.295482 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.295656 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.295866 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.296251 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.296411 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":438,"threads_started":1}
I20260620 15:20:58.296802 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.286721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37575 (local address 127.0.0.1:38728)
0620 15:20:58.287637 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:58.287674 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:58.287799 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:58.288808 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:58.288818 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:58.288854 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:20:58.289205 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.289222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.291627 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.291636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.294064 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.294074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.295821 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.295827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.296016 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.296048 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:20:58.296209 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 15:20:58.296411 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":523,"threads_started":1}
I20260620 15:20:58.300287 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.288049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43321 (local address 127.0.0.1:36614)
0620 15:20:58.288578 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:58.288598 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:58.288729 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:58.293173 (+ 4444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:58.293177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:58.293203 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:20:58.293651 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.293665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.295739 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.295746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.297766 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.297778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.299512 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.299518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.299692 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.299726 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:20:58.299825 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:20:58.299939 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":21,"thread_start_us":256,"threads_started":1}
I20260620 15:20:58.300420 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.288401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.0.97.65:43321)
0620 15:20:58.292627 (+ 4226us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.292634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.292658 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:20:58.292749 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.292755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.292946 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.293869 (+ 923us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:20:58.294122 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.295578 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.297911 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.299312 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.299442 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.299655 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.299881 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.300053 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4043,"thread_start_us":4288,"threads_started":1}
I20260620 15:20:58.301378 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.289360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:38935 (local address 127.0.0.1:55634)
0620 15:20:58.292176 (+ 2816us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:58.292202 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:58.292326 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:58.292992 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:58.292998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:58.293031 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:20:58.293485 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.293497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.295535 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.295542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.298381 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.298394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.300426 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.300433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.300760 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.300803 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 15:20:58.300899 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:20:58.301014 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2657,"mutex_wait_us":28,"thread_start_us":2495,"threads_started":1}
I20260620 15:20:58.300976 17801 tablet_service.cc:1467] Processing CreateTablet for tablet b9622d89f06249e391eef7227f3d7b81 (DEFAULT_TABLE table=test-table [id=bd33884303ce4d158a0055743954c6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:20:58.301674 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.288940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.0.97.67:38935)
0620 15:20:58.289761 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.289773 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.292459 (+ 2686us) server_negotiation.cc:407] Connection header received
0620 15:20:58.292574 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.292580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.292807 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.292988 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:20:58.293705 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.295372 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.298551 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.299926 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.300056 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.300264 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.300979 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.301137 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":521,"threads_started":1}
I20260620 15:20:58.302500 17801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9622d89f06249e391eef7227f3d7b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:20:58.304085 17667 tablet_service.cc:1467] Processing CreateTablet for tablet b9622d89f06249e391eef7227f3d7b81 (DEFAULT_TABLE table=test-table [id=bd33884303ce4d158a0055743954c6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:20:58.305487 17667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9622d89f06249e391eef7227f3d7b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:20:58.305225 17934 tablet_service.cc:1467] Processing CreateTablet for tablet b9622d89f06249e391eef7227f3d7b81 (DEFAULT_TABLE table=test-table [id=bd33884303ce4d158a0055743954c6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:20:58.306514 17934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9622d89f06249e391eef7227f3d7b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:20:58.314255 18017 tablet_bootstrap.cc:492] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: Bootstrap starting.
I20260620 15:20:58.317296 18017 tablet_bootstrap.cc:654] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: Neither blocks nor log segments found. Creating new log.
I20260620 15:20:58.318259 18017 log.cc:824] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:20:58.321611 18020 tablet_bootstrap.cc:492] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: Bootstrap starting.
I20260620 15:20:58.323092 18019 tablet_bootstrap.cc:492] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: Bootstrap starting.
I20260620 15:20:58.325788 18017 tablet_bootstrap.cc:492] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: No bootstrap required, opened a new log
I20260620 15:20:58.326050 18017 ts_tablet_manager.cc:1397] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 15:20:58.326180 18019 tablet_bootstrap.cc:654] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: Neither blocks nor log segments found. Creating new log.
I20260620 15:20:58.326303 18020 tablet_bootstrap.cc:654] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: Neither blocks nor log segments found. Creating new log.
I20260620 15:20:58.327165 18019 log.cc:824] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: Log is configured to *not* fsync() on all Append() calls
I20260620 15:20:58.327472 18020 log.cc:824] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:20:58.329845 18019 tablet_bootstrap.cc:492] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: No bootstrap required, opened a new log
I20260620 15:20:58.330086 18019 ts_tablet_manager.cc:1397] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260620 15:20:58.330904 18020 tablet_bootstrap.cc:492] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: No bootstrap required, opened a new log
I20260620 15:20:58.331140 18020 ts_tablet_manager.cc:1397] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 15:20:58.336403 18017 raft_consensus.cc:348] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 } } peers { permanent_uuid: "0b1c496fa7054e48976412e38b0de735" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43321 } } peers { permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 } }
I20260620 15:20:58.336838 18017 raft_consensus.cc:374] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:20:58.336933 18017 raft_consensus.cc:729] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3b68df98d004936b1b4ca279e1fced5, State: Initialized, Role: FOLLOWER
I20260620 15:20:58.336709 18019 raft_consensus.cc:348] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 } } peers { permanent_uuid: "0b1c496fa7054e48976412e38b0de735" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43321 } } peers { permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 } }
I20260620 15:20:58.337105 18019 raft_consensus.cc:374] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:20:58.337498 18019 raft_consensus.cc:729] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b1c496fa7054e48976412e38b0de735, State: Initialized, Role: FOLLOWER
I20260620 15:20:58.337487 18017 consensus_queue.cc:260] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 } } peers { permanent_uuid: "0b1c496fa7054e48976412e38b0de735" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43321 } } peers { permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 } }
I20260620 15:20:58.338187 18019 consensus_queue.cc:260] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 } } peers { permanent_uuid: "0b1c496fa7054e48976412e38b0de735" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43321 } } peers { permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 } }
I20260620 15:20:58.338539 17866 heartbeater.cc:503] Master 127.0.97.126:41515 was elected leader, sending a full tablet report...
I20260620 15:20:58.339121 18017 ts_tablet_manager.cc:1428] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 15:20:58.340773 18019 ts_tablet_manager.cc:1428] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260620 15:20:58.341202 18020 raft_consensus.cc:348] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 } } peers { permanent_uuid: "0b1c496fa7054e48976412e38b0de735" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43321 } } peers { permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 } }
I20260620 15:20:58.341617 18020 raft_consensus.cc:374] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:20:58.341715 18020 raft_consensus.cc:729] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15be588632c443a18ab878fc9b295e4b, State: Initialized, Role: FOLLOWER
I20260620 15:20:58.342262 18020 consensus_queue.cc:260] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 } } peers { permanent_uuid: "0b1c496fa7054e48976412e38b0de735" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43321 } } peers { permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 } }
I20260620 15:20:58.343281 17999 heartbeater.cc:503] Master 127.0.97.126:41515 was elected leader, sending a full tablet report...
I20260620 15:20:58.343812 18020 ts_tablet_manager.cc:1428] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 15:20:58.372074 18024 raft_consensus.cc:482] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:20:58.372313 18024 raft_consensus.cc:504] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 } } peers { permanent_uuid: "0b1c496fa7054e48976412e38b0de735" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43321 } } peers { permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 } }
I20260620 15:20:58.373962 18024 leader_election.cc:288] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3b68df98d004936b1b4ca279e1fced5 (127.0.97.66:37575), 15be588632c443a18ab878fc9b295e4b (127.0.97.67:38935)
I20260620 15:20:58.381068 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.374125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37575 (local address 127.0.97.65:47641)
0620 15:20:58.374655 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:58.374671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:58.374845 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:58.375334 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:58.375337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:58.375362 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:20:58.375812 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.375828 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.377027 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.377032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.378670 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.378682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.380273 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.380282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.380471 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.380536 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 15:20:58.380655 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 15:20:58.380807 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":279,"threads_started":1}
I20260620 15:20:58.381134 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.374279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47641 (local address 127.0.97.66:37575)
0620 15:20:58.374510 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.374516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.374790 (+ 274us) server_negotiation.cc:407] Connection header received
0620 15:20:58.375027 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.375033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.375114 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.375177 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:20:58.375959 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.376898 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.378846 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.380059 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.380177 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.380250 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.380697 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.380841 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:20:58.381656 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.374892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:45655 (local address 127.0.97.67:38935)
0620 15:20:58.375073 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.375079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.375459 (+ 380us) server_negotiation.cc:407] Connection header received
0620 15:20:58.375647 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.375652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.375727 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.375835 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:20:58.376415 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.377312 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.379470 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.380561 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.380671 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.380763 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.381240 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.381383 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:20:58.382054 17821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9622d89f06249e391eef7227f3d7b81" candidate_uuid: "0b1c496fa7054e48976412e38b0de735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b68df98d004936b1b4ca279e1fced5" is_pre_election: true
I20260620 15:20:58.382267 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.374766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:38935 (local address 127.0.97.65:45655)
0620 15:20:58.375354 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 15:20:58.375372 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:20:58.375519 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:20:58.375844 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:20:58.375847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:20:58.375865 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:20:58.376257 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.376268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.377469 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.377473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.379320 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:20:58.379335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.380785 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.380793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:20:58.380989 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.381026 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:20:58.381848 (+ 822us) client_negotiation.cc:240] Negotiation successful
0620 15:20:58.381962 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"spinlock_wait_cycles":18176,"thread_start_us":298,"threads_started":1}
I20260620 15:20:58.382586 17821 raft_consensus.cc:2454] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b1c496fa7054e48976412e38b0de735 in term 0.
I20260620 15:20:58.383239 17623 leader_election.cc:302] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b1c496fa7054e48976412e38b0de735, d3b68df98d004936b1b4ca279e1fced5; no voters:
I20260620 15:20:58.383252 17954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9622d89f06249e391eef7227f3d7b81" candidate_uuid: "0b1c496fa7054e48976412e38b0de735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15be588632c443a18ab878fc9b295e4b" is_pre_election: true
I20260620 15:20:58.383636 18024 raft_consensus.cc:2781] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:20:58.383852 18024 raft_consensus.cc:482] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:20:58.383941 18024 raft_consensus.cc:3037] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:20:58.383875 17954 raft_consensus.cc:2454] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b1c496fa7054e48976412e38b0de735 in term 0.
I20260620 15:20:58.386291 18024 raft_consensus.cc:504] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 } } peers { permanent_uuid: "0b1c496fa7054e48976412e38b0de735" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43321 } } peers { permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 } }
I20260620 15:20:58.386885 18024 leader_election.cc:288] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [CANDIDATE]: Term 1 election: Requested vote from peers d3b68df98d004936b1b4ca279e1fced5 (127.0.97.66:37575), 15be588632c443a18ab878fc9b295e4b (127.0.97.67:38935)
I20260620 15:20:58.387317 17821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9622d89f06249e391eef7227f3d7b81" candidate_uuid: "0b1c496fa7054e48976412e38b0de735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3b68df98d004936b1b4ca279e1fced5"
I20260620 15:20:58.387396 17954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9622d89f06249e391eef7227f3d7b81" candidate_uuid: "0b1c496fa7054e48976412e38b0de735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15be588632c443a18ab878fc9b295e4b"
I20260620 15:20:58.387521 17821 raft_consensus.cc:3037] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:20:58.387570 17954 raft_consensus.cc:3037] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:20:58.390838 17954 raft_consensus.cc:2454] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b1c496fa7054e48976412e38b0de735 in term 1.
I20260620 15:20:58.390838 17821 raft_consensus.cc:2454] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b1c496fa7054e48976412e38b0de735 in term 1.
I20260620 15:20:58.391245 17623 leader_election.cc:302] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b1c496fa7054e48976412e38b0de735, d3b68df98d004936b1b4ca279e1fced5; no voters:
I20260620 15:20:58.391558 18024 raft_consensus.cc:2781] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:20:58.392004 18024 raft_consensus.cc:686] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 1 LEADER]: Becoming Leader. State: Replica: 0b1c496fa7054e48976412e38b0de735, State: Running, Role: LEADER
I20260620 15:20:58.392477 18024 consensus_queue.cc:237] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 } } peers { permanent_uuid: "0b1c496fa7054e48976412e38b0de735" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43321 } } peers { permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 } }
I20260620 15:20:58.397058 17541 catalog_manager.cc:5495] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b1c496fa7054e48976412e38b0de735 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "0b1c496fa7054e48976412e38b0de735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b1c496fa7054e48976412e38b0de735" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 } health_report { overall_health: UNKNOWN } } }
I20260620 15:20:58.424180 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.418509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.0.97.65:43321)
0620 15:20:58.418726 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.418731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.418748 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:20:58.418831 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.418835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.418921 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.419054 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:20:58.419431 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.420186 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.421689 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.422103 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.422205 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.422721 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:20:58.423422 (+ 701us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:20:58.423597 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.423705 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.423804 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 15:20:58.432288 17954 raft_consensus.cc:1261] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b [term 1 FOLLOWER]: Refusing update from remote peer 0b1c496fa7054e48976412e38b0de735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:20:58.432303 17821 raft_consensus.cc:1261] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5 [term 1 FOLLOWER]: Refusing update from remote peer 0b1c496fa7054e48976412e38b0de735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:20:58.433166 18028 consensus_queue.cc:1035] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3b68df98d004936b1b4ca279e1fced5" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:20:58.433525 18024 consensus_queue.cc:1035] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15be588632c443a18ab878fc9b295e4b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:20:58.443701 18031 mvcc.cc:205] Tried to move back new op lower bound from 7298944444129738752 to 7298944443979345920. Current Snapshot: MvccSnapshot[applied={T|T < 7298944444129738752}]
I20260620 15:20:58.459385 17542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:20:58.459646 17542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:20:58.463323 17542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 212284d256404d98a592dbdeac07a669: Sending DeleteTablet for 3 replicas of tablet b9622d89f06249e391eef7227f3d7b81
I20260620 15:20:58.464341 17801 tablet_service.cc:1514] Processing DeleteTablet for tablet b9622d89f06249e391eef7227f3d7b81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:20:58 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:38728
I20260620 15:20:58.464607 17667 tablet_service.cc:1514] Processing DeleteTablet for tablet b9622d89f06249e391eef7227f3d7b81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:20:58 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:36614
I20260620 15:20:58.464831 17934 tablet_service.cc:1514] Processing DeleteTablet for tablet b9622d89f06249e391eef7227f3d7b81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:20:58 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:55634
I20260620 15:20:58.467074 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 17604
I20260620 15:20:58.482792 18042 tablet_replica.cc:330] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: stopping tablet replica
I20260620 15:20:58.483160 18042 raft_consensus.cc:2229] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:20:58.483729 18042 raft_consensus.cc:2258] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:20:58.486855 18042 ts_tablet_manager.cc:1905] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:20:58.493822 18042 ts_tablet_manager.cc:1918] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:20:58.494007 18042 log.cc:1196] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/b9622d89f06249e391eef7227f3d7b81
I20260620 15:20:58.494680 18042 ts_tablet_manager.cc:1939] T b9622d89f06249e391eef7227f3d7b81 P 0b1c496fa7054e48976412e38b0de735: Deleting consensus metadata
I20260620 15:20:58.496286 18043 tablet_replica.cc:330] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: stopping tablet replica
I20260620 15:20:58.496714 18043 raft_consensus.cc:2229] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:20:58.496847 17530 catalog_manager.cc:4841] TS 0b1c496fa7054e48976412e38b0de735 (127.0.97.65:43321): tablet b9622d89f06249e391eef7227f3d7b81 (table test-table [id=bd33884303ce4d158a0055743954c6b7]) successfully deleted
I20260620 15:20:58.497205 18043 raft_consensus.cc:2258] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:20:58.497454 18041 tablet_replica.cc:330] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: stopping tablet replica
I20260620 15:20:58.497850 18041 raft_consensus.cc:2229] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:20:58.498257 18041 raft_consensus.cc:2258] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:20:58.499814 18041 ts_tablet_manager.cc:1905] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:20:58.500056 18043 ts_tablet_manager.cc:1905] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:20:58.504379 18041 ts_tablet_manager.cc:1918] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:20:58.504551 18041 log.cc:1196] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/b9622d89f06249e391eef7227f3d7b81
I20260620 15:20:58.505085 18041 ts_tablet_manager.cc:1939] T b9622d89f06249e391eef7227f3d7b81 P d3b68df98d004936b1b4ca279e1fced5: Deleting consensus metadata
I20260620 15:20:58.506770 18043 ts_tablet_manager.cc:1918] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:20:58.506901 18043 log.cc:1196] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/b9622d89f06249e391eef7227f3d7b81
I20260620 15:20:58.506897 17530 catalog_manager.cc:4841] TS d3b68df98d004936b1b4ca279e1fced5 (127.0.97.66:37575): tablet b9622d89f06249e391eef7227f3d7b81 (table test-table [id=bd33884303ce4d158a0055743954c6b7]) successfully deleted
I20260620 15:20:58.507405 18043 ts_tablet_manager.cc:1939] T b9622d89f06249e391eef7227f3d7b81 P 15be588632c443a18ab878fc9b295e4b: Deleting consensus metadata
Jun 20 15:20:58 dist-test-slave-7s64 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:20:58.509001 17530 catalog_manager.cc:4841] TS 15be588632c443a18ab878fc9b295e4b (127.0.97.67:38935): tablet b9622d89f06249e391eef7227f3d7b81 (table test-table [id=bd33884303ce4d158a0055743954c6b7]) successfully deleted
I20260620 15:20:58.516742 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.468584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.0.97.65:43321)
0620 15:20:58.468796 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.468805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.490232 (+ 21427us) server_negotiation.cc:407] Connection header received
0620 15:20:58.490599 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.490605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.490686 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.490769 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:58.492594 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.493886 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.497563 (+ 3677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.498255 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.498349 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.510049 (+ 11700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:58.510064 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:58.510081 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:58.510135 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:58.513970 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:58.514611 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:58.514619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:58.514623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:58.514805 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:58.515179 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:58.515185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:58.515190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:58.515599 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:58.515897 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.516090 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.516327 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 15:20:58.517405 17727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:58.621299 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 17604
I20260620 15:20:58.640391 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 17738
Jun 20 15:20:58 dist-test-slave-7s64 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:20:58.654614 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.641218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.0.97.66:37575)
0620 15:20:58.641382 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.641386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.641403 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:20:58.641476 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.641479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.641544 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.641625 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:58.642850 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.643605 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.645130 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.645529 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.645627 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.649476 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:58.649486 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:58.649499 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:58.649529 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:58.652137 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:58.652862 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:58.652868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:58.652871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:58.652990 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:58.653310 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:58.653313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:58.653315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:58.653670 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:58.653918 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.654217 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.654375 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:20:58.655310 17861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:58.742542 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 17738
I20260620 15:20:58.759505 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 17871
Jun 20 15:20:58 dist-test-slave-7s64 krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:20:58.775676 18015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:58.760243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.0.97.67:38935)
0620 15:20:58.760420 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:58.760424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:58.760473 (+ 49us) server_negotiation.cc:407] Connection header received
0620 15:20:58.760738 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:58.760743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:58.760840 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:58.760943 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:58.762264 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.763273 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.764716 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:58.765132 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:58.765232 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:58.769424 (+ 4192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:58.769439 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:58.769456 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:20:58.769505 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:58.773033 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:58.773684 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:58.773691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:58.773694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:58.773850 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:20:58.774177 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:20:58.774180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:20:58.774183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:20:58.774561 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:58.774832 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:58.775057 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 15:20:58.775266 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:20:58.776382 17994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:58.863927 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 17871
I20260620 15:20:58.880976 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 17511
I20260620 15:20:58.882205 17573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:20:58.990571 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 17511
2026-06-20T15:20:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4106 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T15:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:20:59Z Disabled control of system clock
I20260620 15:20:59.063592 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43765
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:20:59.403565 18052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:59.403868 18052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:59.404024 18052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:59.411140 18052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:20:59.411265 18052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:59.411356 18052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:59.411441 18052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:20:59.411528 18052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:20:59.420053 18052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:59.421275 18052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:59.422923 18052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:59.435940 18058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:59.436235 18057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:59.436432 18060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:59.438186 18052 server_base.cc:1034] running on GCE node
I20260620 15:20:59.439087 18052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:59.440897 18052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:59.442137 18052 hybrid_clock.cc:648] HybridClock initialized: now 1781968859442081 us; error 41 us; skew 500 ppm
I20260620 15:20:59.442485 18052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:20:59.445008 18052 webserver.cc:460] Webserver started at http://127.0.97.126:36545/ using document root <none> and password file <none>
I20260620 15:20:59.445678 18052 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:59.445807 18052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:59.446141 18052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:20:59.449002 18052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "7c76db6ceb864ebd8b099f7f4978b919"
format_stamp: "Formatted at 2026-06-20 15:20:59 on dist-test-slave-7s64"
server_key: "5d026e58b3e768be7f009edaea2062b6"
server_key_iv: "a11f91e084b3ed62e3b32fdfe209f921"
server_key_version: "encryptionkey@0"
I20260620 15:20:59.449810 18052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "7c76db6ceb864ebd8b099f7f4978b919"
format_stamp: "Formatted at 2026-06-20 15:20:59 on dist-test-slave-7s64"
server_key: "5d026e58b3e768be7f009edaea2062b6"
server_key_iv: "a11f91e084b3ed62e3b32fdfe209f921"
server_key_version: "encryptionkey@0"
I20260620 15:20:59.454810 18052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 15:20:59.458389 18066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:59.459755 18052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:20:59.459936 18052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "7c76db6ceb864ebd8b099f7f4978b919"
format_stamp: "Formatted at 2026-06-20 15:20:59 on dist-test-slave-7s64"
server_key: "5d026e58b3e768be7f009edaea2062b6"
server_key_iv: "a11f91e084b3ed62e3b32fdfe209f921"
server_key_version: "encryptionkey@0"
I20260620 15:20:59.460147 18052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:59.482801 18052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:59.483625 18052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:59.483906 18052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:59.501997 18052 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43765
I20260620 15:20:59.502018 18117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43765 every 8 connection(s)
I20260620 15:20:59.503379 18052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:20:59.506953 18118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:20:59.512313 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 18052
I20260620 15:20:59.512540 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:20:59.512988 389 external_mini_cluster.cc:1428] Setting key 7728447299cd4294552ab4f0c00a489c
I20260620 15:20:59.517609 18118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919: Bootstrap starting.
I20260620 15:20:59.520613 18118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919: Neither blocks nor log segments found. Creating new log.
I20260620 15:20:59.521533 18118 log.cc:824] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919: Log is configured to *not* fsync() on all Append() calls
I20260620 15:20:59.524189 18118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919: No bootstrap required, opened a new log
I20260620 15:20:59.526435 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:20:59.514854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.0.97.126:43765)
0620 15:20:59.515817 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0620 15:20:59.515827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:20:59.515866 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:20:59.517567 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:20:59.517589 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:20:59.518068 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:20:59.518442 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:20:59.518986 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:59.520240 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:59.521758 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:20:59.522823 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:20:59.523015 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:20:59.524181 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:20:59.524188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:20:59.524201 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:20:59.524842 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:20:59.524918 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:20:59.524941 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:20:59.525065 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:20:59.525247 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 15:20:59.525537 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":349,"threads_started":1}
I20260620 15:20:59.532388 18118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c76db6ceb864ebd8b099f7f4978b919" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43765 } }
I20260620 15:20:59.532796 18118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:20:59.532900 18118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c76db6ceb864ebd8b099f7f4978b919, State: Initialized, Role: FOLLOWER
I20260620 15:20:59.533418 18118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c76db6ceb864ebd8b099f7f4978b919" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43765 } }
I20260620 15:20:59.533641 18118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:20:59.533797 18118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:20:59.533972 18118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:20:59.536048 18118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c76db6ceb864ebd8b099f7f4978b919" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43765 } }
I20260620 15:20:59.536595 18118 leader_election.cc:302] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c76db6ceb864ebd8b099f7f4978b919; no voters:
I20260620 15:20:59.537123 18118 leader_election.cc:288] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:20:59.537269 18123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:20:59.538479 18123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [term 1 LEADER]: Becoming Leader. State: Replica: 7c76db6ceb864ebd8b099f7f4978b919, State: Running, Role: LEADER
I20260620 15:20:59.539016 18123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c76db6ceb864ebd8b099f7f4978b919" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43765 } }
I20260620 15:20:59.539143 18118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:20:59.541667 18124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c76db6ceb864ebd8b099f7f4978b919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c76db6ceb864ebd8b099f7f4978b919" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43765 } } }
I20260620 15:20:59.542035 18124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [sys.catalog]: This master's current role is: LEADER
I20260620 15:20:59.542539 18125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c76db6ceb864ebd8b099f7f4978b919. Latest consensus state: current_term: 1 leader_uuid: "7c76db6ceb864ebd8b099f7f4978b919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c76db6ceb864ebd8b099f7f4978b919" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43765 } } }
I20260620 15:20:59.542838 18125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919 [sys.catalog]: This master's current role is: LEADER
I20260620 15:20:59.543047 18131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:20:59.549573 18131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:20:59.558416 18131 catalog_manager.cc:1347] Generated new cluster ID: 4366189131a340028ce1b54161817e8a
I20260620 15:20:59.558532 18131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:20:59.575614 18131 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:20:59.577394 18131 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:20:59.599368 18131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919: Generated new TSK 0
I20260620 15:20:59.600431 18131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:20:59.606688 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43765
--builtin_ntp_servers=127.0.97.84:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:20:59.926751 18142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:20:59.926999 18142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:20:59.927148 18142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:20:59.933912 18142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:20:59.933992 18142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:20:59.934122 18142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:20:59.942909 18142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:20:59.944005 18142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:20:59.945827 18142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:20:59.958153 18148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:59.958220 18147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:20:59.958199 18150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:20:59.959233 18142 server_base.cc:1034] running on GCE node
I20260620 15:20:59.959830 18142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:20:59.960865 18142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:20:59.962090 18142 hybrid_clock.cc:648] HybridClock initialized: now 1781968859962053 us; error 46 us; skew 500 ppm
I20260620 15:20:59.962399 18142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:20:59.969142 18142 webserver.cc:460] Webserver started at http://127.0.97.65:36561/ using document root <none> and password file <none>
I20260620 15:20:59.969827 18142 fs_manager.cc:359] Metadata directory not provided
I20260620 15:20:59.969960 18142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:20:59.970276 18142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:20:59.973126 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "3c0835b1f1a9418eb7c4322befa9ebde"
format_stamp: "Formatted at 2026-06-20 15:20:59 on dist-test-slave-7s64"
server_key: "af00df39c4976cd371d41f40c1b8b3f9"
server_key_iv: "7b7fe33355972ee331456cd4eb371e7e"
server_key_version: "encryptionkey@0"
I20260620 15:20:59.973922 18142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "3c0835b1f1a9418eb7c4322befa9ebde"
format_stamp: "Formatted at 2026-06-20 15:20:59 on dist-test-slave-7s64"
server_key: "af00df39c4976cd371d41f40c1b8b3f9"
server_key_iv: "7b7fe33355972ee331456cd4eb371e7e"
server_key_version: "encryptionkey@0"
I20260620 15:20:59.978880 18142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:20:59.982451 18156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:59.983793 18142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:20:59.983973 18142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "3c0835b1f1a9418eb7c4322befa9ebde"
format_stamp: "Formatted at 2026-06-20 15:20:59 on dist-test-slave-7s64"
server_key: "af00df39c4976cd371d41f40c1b8b3f9"
server_key_iv: "7b7fe33355972ee331456cd4eb371e7e"
server_key_version: "encryptionkey@0"
I20260620 15:20:59.984203 18142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:20:59.995807 18142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:20:59.996582 18142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:20:59.996852 18142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:20:59.997756 18142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:20:59.999141 18142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:20:59.999225 18142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:20:59.999372 18142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:20:59.999466 18142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:00.032575 18142 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:40863
I20260620 15:21:00.032711 18268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:40863 every 8 connection(s)
I20260620 15:21:00.033878 18142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:21:00.039361 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 18142
I20260620 15:21:00.039566 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:21:00.040046 389 external_mini_cluster.cc:1428] Setting key 852af513eebd46f95bfe356aeb9299d3
I20260620 15:21:00.043095 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43765
--builtin_ntp_servers=127.0.97.84:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:21:00.047320 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.036417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:52899 (local address 127.0.97.126:43765)
0620 15:21:00.036910 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:00.036914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:00.037994 (+ 1080us) server_negotiation.cc:407] Connection header received
0620 15:21:00.039863 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:00.039868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:00.039956 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:00.040068 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:00.041336 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.042315 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.044024 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.044861 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.045017 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:00.046329 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:00.046338 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:00.046346 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:00.046373 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:00.046430 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:00.046450 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:00.046552 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:00.046746 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 15:21:00.046857 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":281,"threads_started":1}
I20260620 15:21:00.047959 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.036972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43765 (local address 127.0.97.65:52899)
0620 15:21:00.037841 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:00.037879 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:00.039582 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:00.040298 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:00.040306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:00.040404 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:00.041152 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.041170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.042475 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.042479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.043867 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.043878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.045060 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.045065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.045261 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:00.046012 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:21:00.046015 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:00.046056 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:21:00.046132 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:00.046548 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:00.046555 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:00.046561 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:21:00.046700 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:21:00.046946 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":293,"threads_started":1}
I20260620 15:21:00.049427 18269 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43765
I20260620 15:21:00.049706 18269 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:00.050374 18269 heartbeater.cc:511] Master 127.0.97.126:43765 requested a full tablet report, sending...
I20260620 15:21:00.052563 18083 ts_manager.cc:194] Registered new tserver with Master: 3c0835b1f1a9418eb7c4322befa9ebde (127.0.97.65:40863)
I20260620 15:21:00.054231 18083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:52899
W20260620 15:21:00.358515 18273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:00.358760 18273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:00.358891 18273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:00.366024 18273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:00.366103 18273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:00.366314 18273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:21:00.374934 18273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:00.376068 18273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:00.377767 18273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:00.390009 18281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:00.390141 18279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:00.390542 18278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:00.391839 18273 server_base.cc:1034] running on GCE node
I20260620 15:21:00.392366 18273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:00.393261 18273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:00.394454 18273 hybrid_clock.cc:648] HybridClock initialized: now 1781968860394392 us; error 57 us; skew 500 ppm
I20260620 15:21:00.394739 18273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:00.396919 18273 webserver.cc:460] Webserver started at http://127.0.97.66:36767/ using document root <none> and password file <none>
I20260620 15:21:00.397570 18273 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:00.397675 18273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:00.398011 18273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:00.400765 18273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "1440d488e96040e4a8e94382a2662887"
format_stamp: "Formatted at 2026-06-20 15:21:00 on dist-test-slave-7s64"
server_key: "33b5049ee24ca940cf54ccd454da0ee6"
server_key_iv: "38dc15ae8a50ec3735142f7d7cc1e5de"
server_key_version: "encryptionkey@0"
I20260620 15:21:00.401507 18273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "1440d488e96040e4a8e94382a2662887"
format_stamp: "Formatted at 2026-06-20 15:21:00 on dist-test-slave-7s64"
server_key: "33b5049ee24ca940cf54ccd454da0ee6"
server_key_iv: "38dc15ae8a50ec3735142f7d7cc1e5de"
server_key_version: "encryptionkey@0"
I20260620 15:21:00.406409 18273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:21:00.409744 18287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:00.411042 18273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:21:00.411204 18273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "1440d488e96040e4a8e94382a2662887"
format_stamp: "Formatted at 2026-06-20 15:21:00 on dist-test-slave-7s64"
server_key: "33b5049ee24ca940cf54ccd454da0ee6"
server_key_iv: "38dc15ae8a50ec3735142f7d7cc1e5de"
server_key_version: "encryptionkey@0"
I20260620 15:21:00.411408 18273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:00.445670 18273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:00.446448 18273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:00.446722 18273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:00.447692 18273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:00.449079 18273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:00.449194 18273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:00.449316 18273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:00.449414 18273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:00.481462 18273 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:44549
I20260620 15:21:00.481551 18399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:44549 every 8 connection(s)
I20260620 15:21:00.482751 18273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:21:00.487833 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 18273
I20260620 15:21:00.488018 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:21:00.488631 389 external_mini_cluster.cc:1428] Setting key 199f2eb4c866836ae57ee6fe7ef024cc
I20260620 15:21:00.491344 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43765
--builtin_ntp_servers=127.0.97.84:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:21:00.496752 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.485116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52121 (local address 127.0.97.126:43765)
0620 15:21:00.485294 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:00.485299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:00.486678 (+ 1379us) server_negotiation.cc:407] Connection header received
0620 15:21:00.488999 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:00.489006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:00.489138 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:00.489277 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:00.490453 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.491242 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.493778 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.494189 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.494289 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:00.495964 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:00.495969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:00.495972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:00.495989 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:00.496022 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:00.496034 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:00.496112 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:00.496367 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 15:21:00.496445 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:21:00.498108 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.485603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43765 (local address 127.0.97.66:52121)
0620 15:21:00.486489 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:00.486533 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:00.488689 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:00.489494 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:00.489529 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:00.489684 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:00.490265 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.490289 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.491424 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.491432 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.493592 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.493610 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.494424 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.494437 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.494776 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:00.495548 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:21:00.495554 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:00.495605 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:21:00.495723 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:00.496142 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:00.496156 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:00.496167 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:21:00.496351 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 15:21:00.496796 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":274,"threads_started":1}
I20260620 15:21:00.499526 18400 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43765
I20260620 15:21:00.499819 18400 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:00.500700 18400 heartbeater.cc:511] Master 127.0.97.126:43765 requested a full tablet report, sending...
I20260620 15:21:00.502275 18083 ts_manager.cc:194] Registered new tserver with Master: 1440d488e96040e4a8e94382a2662887 (127.0.97.66:44549)
I20260620 15:21:00.503509 18083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:52121
W20260620 15:21:00.806116 18403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:00.806349 18403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:00.806506 18403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:00.813436 18403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:00.813524 18403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:00.813674 18403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:21:00.822652 18403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:00.823817 18403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:00.825613 18403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:00.837946 18408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:00.837991 18409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:00.837900 18411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:00.839138 18403 server_base.cc:1034] running on GCE node
I20260620 15:21:00.839756 18403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:00.840691 18403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:00.841970 18403 hybrid_clock.cc:648] HybridClock initialized: now 1781968860841934 us; error 41 us; skew 500 ppm
I20260620 15:21:00.842314 18403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:00.844657 18403 webserver.cc:460] Webserver started at http://127.0.97.67:37019/ using document root <none> and password file <none>
I20260620 15:21:00.845363 18403 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:00.845485 18403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:00.845858 18403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:00.848630 18403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "428dcc5974634c32afc861f32236cf69"
format_stamp: "Formatted at 2026-06-20 15:21:00 on dist-test-slave-7s64"
server_key: "6587e6079ca873ae104397097f609775"
server_key_iv: "c23fb85b514314b41be17f71090a5ed0"
server_key_version: "encryptionkey@0"
I20260620 15:21:00.849375 18403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "428dcc5974634c32afc861f32236cf69"
format_stamp: "Formatted at 2026-06-20 15:21:00 on dist-test-slave-7s64"
server_key: "6587e6079ca873ae104397097f609775"
server_key_iv: "c23fb85b514314b41be17f71090a5ed0"
server_key_version: "encryptionkey@0"
I20260620 15:21:00.853940 18403 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:21:00.857157 18417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:00.858435 18403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:21:00.858613 18403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "428dcc5974634c32afc861f32236cf69"
format_stamp: "Formatted at 2026-06-20 15:21:00 on dist-test-slave-7s64"
server_key: "6587e6079ca873ae104397097f609775"
server_key_iv: "c23fb85b514314b41be17f71090a5ed0"
server_key_version: "encryptionkey@0"
I20260620 15:21:00.858870 18403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:00.887460 18403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:00.888170 18403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:00.888420 18403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:00.889447 18403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:00.891132 18403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:00.891254 18403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:00.891371 18403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:00.891446 18403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:00.923493 18403 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:42497
I20260620 15:21:00.923575 18529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:42497 every 8 connection(s)
I20260620 15:21:00.924743 18403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:21:00.926710 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 18403
I20260620 15:21:00.926893 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:21:00.927403 389 external_mini_cluster.cc:1428] Setting key 4fadcc2db68259843a69bd23554abd5f
I20260620 15:21:00.936852 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.927458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35339 (local address 127.0.97.126:43765)
0620 15:21:00.928405 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:00.928409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:00.929198 (+ 789us) server_negotiation.cc:407] Connection header received
0620 15:21:00.930994 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:00.930998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:00.931061 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:00.931142 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:00.932124 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.932942 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.934389 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.934843 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.934958 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:00.936165 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:00.936169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:00.936173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:00.936188 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:00.936217 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:00.936229 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:00.936299 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:00.936543 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 15:21:00.936624 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840}
I20260620 15:21:00.937755 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.927957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43765 (local address 127.0.97.67:35339)
0620 15:21:00.929018 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:00.929078 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:00.930777 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:00.931366 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:00.931373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:00.931452 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:00.931981 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.931996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.933076 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.933079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.934247 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.934257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.935034 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.935038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.935228 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:00.935884 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:21:00.935887 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:00.935924 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:21:00.935991 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:00.936317 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:00.936325 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:00.936330 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:21:00.936472 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 15:21:00.936790 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":347,"threads_started":1}
I20260620 15:21:00.939067 18530 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43765
I20260620 15:21:00.939419 18530 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:00.940246 18530 heartbeater.cc:511] Master 127.0.97.126:43765 requested a full tablet report, sending...
I20260620 15:21:00.942014 18083 ts_manager.cc:194] Registered new tserver with Master: 428dcc5974634c32afc861f32236cf69 (127.0.97.67:42497)
I20260620 15:21:00.942827 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:21:00.943073 18083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:35339
I20260620 15:21:00.953495 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.947824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.0.97.126:43765)
0620 15:21:00.947979 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:00.947984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:00.948432 (+ 448us) server_negotiation.cc:407] Connection header received
0620 15:21:00.948653 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:00.948657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:00.948728 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:00.948815 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:00.949247 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.949904 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.951308 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.951742 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.951856 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:00.952873 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:00.952878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:00.952881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:00.952899 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:00.952931 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:00.952944 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:00.953023 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:00.953160 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 15:21:00.953241 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:21:00.957276 18083 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:21:00.964182 18083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:21:00.999893 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.975577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:42497 (local address 127.0.0.1:60112)
0620 15:21:00.976529 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:00.976583 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:00.976786 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:00.991888 (+ 15102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:00.991896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:00.991926 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:00.992377 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.992390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.994385 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.994392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.997314 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.997327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.998765 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.998775 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.999002 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:00.999076 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 15:21:00.999214 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 15:21:00.999414 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":6,"thread_start_us":17700,"threads_started":1}
I20260620 15:21:01.000267 18546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.975720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:40863 (local address 127.0.0.1:47958)
0620 15:21:00.991730 (+ 16010us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:00.991757 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:00.991887 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:00.993234 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:00.993242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:00.993287 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:00.993859 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.993873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.995473 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.995478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.997333 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.997351 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.999464 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.999472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.999671 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:00.999717 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 15:21:00.999806 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 15:21:00.999931 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15904,"mutex_wait_us":84,"thread_start_us":1012,"threads_started":1}
I20260620 15:21:01.001443 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.975826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.0.97.67:42497)
0620 15:21:00.991132 (+ 15306us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:00.991145 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:00.991191 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:21:00.991319 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:00.991332 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:00.991630 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:00.991871 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:00.992613 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.994196 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.997504 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.998517 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.000474 (+ 1957us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.000800 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:01.000870 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 15:21:01.001039 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15071,"thread_start_us":14888,"threads_started":1}
I20260620 15:21:01.001459 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.976399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.0.97.65:40863)
0620 15:21:00.992347 (+ 15948us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:00.992356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:00.992383 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:21:00.992478 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:00.992488 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:00.992778 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:00.992958 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:00.994045 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.995311 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.997586 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.998970 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.999086 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:00.999332 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:01.000588 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0620 15:21:01.001120 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15774,"thread_start_us":16488,"threads_started":1}
I20260620 15:21:01.002599 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.977673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44549 (local address 127.0.0.1:60596)
0620 15:21:00.993612 (+ 15939us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:00.993629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:00.993761 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:00.994673 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:00.994679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:00.994707 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:00.995218 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.995232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.997237 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.997243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:00.999260 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:00.999274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.001954 (+ 2680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.001958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:01.002068 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.002087 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 15:21:01.002199 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 15:21:01.002322 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15806,"thread_start_us":13838,"threads_started":1}
I20260620 15:21:01.002763 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:00.978109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.0.97.66:44549)
0620 15:21:00.991398 (+ 13289us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:00.991404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:00.993748 (+ 2344us) server_negotiation.cc:407] Connection header received
0620 15:21:00.993905 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:00.993914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:00.994149 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:00.994376 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:00.995439 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:00.997053 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:00.999496 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.001443 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.001576 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.001823 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:01.002231 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 15:21:01.002387 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13133,"thread_start_us":12985,"threads_started":1}
I20260620 15:21:01.005429 18465 tablet_service.cc:1467] Processing CreateTablet for tablet 26d2da62842d46d49a0231d29ccb09ae (DEFAULT_TABLE table=test-table [id=bd474f81c0084ae9ab7eba16c64058be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:01.005741 18335 tablet_service.cc:1467] Processing CreateTablet for tablet 26d2da62842d46d49a0231d29ccb09ae (DEFAULT_TABLE table=test-table [id=bd474f81c0084ae9ab7eba16c64058be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:01.007030 18465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d2da62842d46d49a0231d29ccb09ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:01.007030 18335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d2da62842d46d49a0231d29ccb09ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:01.009322 18204 tablet_service.cc:1467] Processing CreateTablet for tablet 26d2da62842d46d49a0231d29ccb09ae (DEFAULT_TABLE table=test-table [id=bd474f81c0084ae9ab7eba16c64058be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:01.010548 18204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d2da62842d46d49a0231d29ccb09ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:01.019529 18548 tablet_bootstrap.cc:492] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: Bootstrap starting.
I20260620 15:21:01.022791 18548 tablet_bootstrap.cc:654] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:01.023641 18549 tablet_bootstrap.cc:492] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: Bootstrap starting.
I20260620 15:21:01.025058 18550 tablet_bootstrap.cc:492] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: Bootstrap starting.
I20260620 15:21:01.025928 18548 log.cc:824] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:01.027475 18549 tablet_bootstrap.cc:654] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:01.028072 18550 tablet_bootstrap.cc:654] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:01.028695 18549 log.cc:824] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:01.029109 18550 log.cc:824] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:01.029186 18548 tablet_bootstrap.cc:492] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: No bootstrap required, opened a new log
I20260620 15:21:01.029485 18548 ts_tablet_manager.cc:1397] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 15:21:01.031533 18549 tablet_bootstrap.cc:492] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: No bootstrap required, opened a new log
I20260620 15:21:01.031787 18549 ts_tablet_manager.cc:1397] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 15:21:01.033113 18550 tablet_bootstrap.cc:492] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: No bootstrap required, opened a new log
I20260620 15:21:01.033438 18550 ts_tablet_manager.cc:1397] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 15:21:01.040622 18548 raft_consensus.cc:348] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428dcc5974634c32afc861f32236cf69" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42497 } } peers { permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 } } peers { permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 } }
I20260620 15:21:01.041046 18548 raft_consensus.cc:374] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:01.041148 18548 raft_consensus.cc:729] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428dcc5974634c32afc861f32236cf69, State: Initialized, Role: FOLLOWER
I20260620 15:21:01.041869 18548 consensus_queue.cc:260] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428dcc5974634c32afc861f32236cf69" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42497 } } peers { permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 } } peers { permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 } }
I20260620 15:21:01.041924 18549 raft_consensus.cc:348] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428dcc5974634c32afc861f32236cf69" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42497 } } peers { permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 } } peers { permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 } }
I20260620 15:21:01.042273 18549 raft_consensus.cc:374] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:01.042367 18549 raft_consensus.cc:729] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1440d488e96040e4a8e94382a2662887, State: Initialized, Role: FOLLOWER
I20260620 15:21:01.042862 18549 consensus_queue.cc:260] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428dcc5974634c32afc861f32236cf69" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42497 } } peers { permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 } } peers { permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 } }
I20260620 15:21:01.046180 18550 raft_consensus.cc:348] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428dcc5974634c32afc861f32236cf69" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42497 } } peers { permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 } } peers { permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 } }
I20260620 15:21:01.046730 18550 raft_consensus.cc:374] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:01.046900 18550 raft_consensus.cc:729] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c0835b1f1a9418eb7c4322befa9ebde, State: Initialized, Role: FOLLOWER
I20260620 15:21:01.047758 18550 consensus_queue.cc:260] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428dcc5974634c32afc861f32236cf69" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42497 } } peers { permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 } } peers { permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 } }
I20260620 15:21:01.051314 18548 ts_tablet_manager.cc:1428] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: Time spent starting tablet: real 0.022s user 0.008s sys 0.010s
I20260620 15:21:01.051729 18530 heartbeater.cc:503] Master 127.0.97.126:43765 was elected leader, sending a full tablet report...
I20260620 15:21:01.051904 18400 heartbeater.cc:503] Master 127.0.97.126:43765 was elected leader, sending a full tablet report...
I20260620 15:21:01.052556 18549 ts_tablet_manager.cc:1428] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: Time spent starting tablet: real 0.021s user 0.008s sys 0.013s
I20260620 15:21:01.054762 18269 heartbeater.cc:503] Master 127.0.97.126:43765 was elected leader, sending a full tablet report...
I20260620 15:21:01.055455 18550 ts_tablet_manager.cc:1428] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s
W20260620 15:21:01.176410 18531 tablet.cc:2369] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:21:01.234573 18401 tablet.cc:2369] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:21:01.265631 18554 raft_consensus.cc:482] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:21:01.265949 18554 raft_consensus.cc:504] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428dcc5974634c32afc861f32236cf69" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42497 } } peers { permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 } } peers { permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 } }
I20260620 15:21:01.267658 18554 leader_election.cc:288] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c0835b1f1a9418eb7c4322befa9ebde (127.0.97.65:40863), 1440d488e96040e4a8e94382a2662887 (127.0.97.66:44549)
I20260620 15:21:01.273191 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:01.268011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:40863 (local address 127.0.97.67:35463)
0620 15:21:01.268194 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:01.268233 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:01.268377 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:01.268772 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:01.268777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:01.268809 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:01.269025 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:01.269034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.270040 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.270048 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:01.271348 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:01.271362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.272445 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.272451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:01.272649 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.272673 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:21:01.272779 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 15:21:01.272911 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:21:01.273192 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:01.268129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35463 (local address 127.0.97.65:40863)
0620 15:21:01.268365 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:01.268371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:01.268390 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:21:01.268561 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:01.268565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:01.268640 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:01.268743 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:01.269189 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.269891 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.271513 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.272283 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.272432 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.272541 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:01.272813 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:21:01.272933 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 15:21:01.274204 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:01.268461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44549 (local address 127.0.97.67:59725)
0620 15:21:01.269238 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:01.269252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:01.269389 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:01.269713 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:01.269718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:01.269739 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:01.270054 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:01.270062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.271068 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.271074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:01.272394 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:01.272404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.273609 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.273617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:01.273814 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.273846 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:21:01.273936 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 15:21:01.274013 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":455,"threads_started":1}
I20260620 15:21:01.274369 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:01.268694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59725 (local address 127.0.97.66:44549)
0620 15:21:01.268907 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:01.268915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:01.269348 (+ 433us) server_negotiation.cc:407] Connection header received
0620 15:21:01.269536 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:01.269541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:01.269599 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:01.269702 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:01.270227 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.270908 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.272596 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.273441 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.273578 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.273651 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:01.273991 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 15:21:01.274124 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:21:01.274402 18224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d2da62842d46d49a0231d29ccb09ae" candidate_uuid: "428dcc5974634c32afc861f32236cf69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" is_pre_election: true
I20260620 15:21:01.274995 18224 raft_consensus.cc:2454] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428dcc5974634c32afc861f32236cf69 in term 0.
I20260620 15:21:01.275259 18355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d2da62842d46d49a0231d29ccb09ae" candidate_uuid: "428dcc5974634c32afc861f32236cf69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1440d488e96040e4a8e94382a2662887" is_pre_election: true
I20260620 15:21:01.275655 18419 leader_election.cc:302] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0835b1f1a9418eb7c4322befa9ebde, 428dcc5974634c32afc861f32236cf69; no voters:
I20260620 15:21:01.275812 18355 raft_consensus.cc:2454] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428dcc5974634c32afc861f32236cf69 in term 0.
I20260620 15:21:01.276018 18554 raft_consensus.cc:2781] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:21:01.276149 18554 raft_consensus.cc:482] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:21:01.276261 18554 raft_consensus.cc:3037] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:01.278437 18554 raft_consensus.cc:504] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428dcc5974634c32afc861f32236cf69" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42497 } } peers { permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 } } peers { permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 } }
I20260620 15:21:01.278980 18554 leader_election.cc:288] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [CANDIDATE]: Term 1 election: Requested vote from peers 3c0835b1f1a9418eb7c4322befa9ebde (127.0.97.65:40863), 1440d488e96040e4a8e94382a2662887 (127.0.97.66:44549)
I20260620 15:21:01.279359 18224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d2da62842d46d49a0231d29ccb09ae" candidate_uuid: "428dcc5974634c32afc861f32236cf69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c0835b1f1a9418eb7c4322befa9ebde"
I20260620 15:21:01.279421 18355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d2da62842d46d49a0231d29ccb09ae" candidate_uuid: "428dcc5974634c32afc861f32236cf69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1440d488e96040e4a8e94382a2662887"
I20260620 15:21:01.279539 18224 raft_consensus.cc:3037] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:01.279626 18355 raft_consensus.cc:3037] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:01.281796 18224 raft_consensus.cc:2454] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428dcc5974634c32afc861f32236cf69 in term 1.
I20260620 15:21:01.281842 18355 raft_consensus.cc:2454] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428dcc5974634c32afc861f32236cf69 in term 1.
I20260620 15:21:01.282145 18419 leader_election.cc:302] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c0835b1f1a9418eb7c4322befa9ebde, 428dcc5974634c32afc861f32236cf69; no voters:
I20260620 15:21:01.282407 18554 raft_consensus.cc:2781] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:01.282871 18554 raft_consensus.cc:686] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 1 LEADER]: Becoming Leader. State: Replica: 428dcc5974634c32afc861f32236cf69, State: Running, Role: LEADER
I20260620 15:21:01.283315 18554 consensus_queue.cc:237] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428dcc5974634c32afc861f32236cf69" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42497 } } peers { permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 } } peers { permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 } }
W20260620 15:21:01.286113 18270 tablet.cc:2369] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:21:01.288662 18083 catalog_manager.cc:5495] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 428dcc5974634c32afc861f32236cf69 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "428dcc5974634c32afc861f32236cf69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428dcc5974634c32afc861f32236cf69" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 } health_report { overall_health: UNKNOWN } } }
I20260620 15:21:01.368280 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:01.362571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.0.97.67:42497)
0620 15:21:01.362786 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:01.362792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:01.362911 (+ 119us) server_negotiation.cc:407] Connection header received
0620 15:21:01.363155 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:01.363162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:01.363262 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:01.363426 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:01.363802 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.364680 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.366142 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.366572 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.366732 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.367198 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:01.367716 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:01.367878 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:01.367929 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 15:21:01.368003 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 15:21:01.375720 18355 raft_consensus.cc:1261] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887 [term 1 FOLLOWER]: Refusing update from remote peer 428dcc5974634c32afc861f32236cf69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:01.375794 18224 raft_consensus.cc:1261] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde [term 1 FOLLOWER]: Refusing update from remote peer 428dcc5974634c32afc861f32236cf69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:01.376704 18558 consensus_queue.cc:1035] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1440d488e96040e4a8e94382a2662887" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:01.377064 18554 consensus_queue.cc:1035] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c0835b1f1a9418eb7c4322befa9ebde" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:01.389240 18562 mvcc.cc:205] Tried to move back new op lower bound from 7298944456186236928 to 7298944455819735040. Current Snapshot: MvccSnapshot[applied={T|T < 7298944456186236928}]
I20260620 15:21:01.399210 18083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:21:01.399433 18083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:21:01.402838 18083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c76db6ceb864ebd8b099f7f4978b919: Sending DeleteTablet for 3 replicas of tablet 26d2da62842d46d49a0231d29ccb09ae
I20260620 15:21:01.404048 18465 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d2da62842d46d49a0231d29ccb09ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:01 UTC) from {username='slave'} at 127.0.0.1:60112
I20260620 15:21:01.404058 18335 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d2da62842d46d49a0231d29ccb09ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:01 UTC) from {username='slave'} at 127.0.0.1:60596
I20260620 15:21:01.404330 18204 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d2da62842d46d49a0231d29ccb09ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:01 UTC) from {username='slave'} at 127.0.0.1:47958
I20260620 15:21:01.404915 18571 tablet_replica.cc:330] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: stopping tablet replica
I20260620 15:21:01.405301 18571 raft_consensus.cc:2229] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:21:01.405550 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 18142
I20260620 15:21:01.405766 18571 raft_consensus.cc:2258] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:01.414717 18571 ts_tablet_manager.cc:1905] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:01.421523 18571 ts_tablet_manager.cc:1918] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:01.421737 18571 log.cc:1196] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/26d2da62842d46d49a0231d29ccb09ae
I20260620 15:21:01.422303 18571 ts_tablet_manager.cc:1939] T 26d2da62842d46d49a0231d29ccb09ae P 1440d488e96040e4a8e94382a2662887: Deleting consensus metadata
I20260620 15:21:01.423156 18572 tablet_replica.cc:330] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: stopping tablet replica
I20260620 15:21:01.423727 18573 tablet_replica.cc:330] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: stopping tablet replica
I20260620 15:21:01.423979 18068 catalog_manager.cc:4841] TS 1440d488e96040e4a8e94382a2662887 (127.0.97.66:44549): tablet 26d2da62842d46d49a0231d29ccb09ae (table test-table [id=bd474f81c0084ae9ab7eba16c64058be]) successfully deleted
I20260620 15:21:01.424391 18572 raft_consensus.cc:2229] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:21:01.424427 18573 raft_consensus.cc:2229] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:21:01.424901 18573 raft_consensus.cc:2258] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:01.425020 18572 raft_consensus.cc:2258] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:01.426806 18572 ts_tablet_manager.cc:1905] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:01.427809 18573 ts_tablet_manager.cc:1905] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:01.432768 18573 ts_tablet_manager.cc:1918] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:01.432972 18573 log.cc:1196] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/26d2da62842d46d49a0231d29ccb09ae
I20260620 15:21:01.433741 18545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:01.406877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.0.97.65:40863)
0620 15:21:01.407051 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:01.407058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:01.414655 (+ 7597us) server_negotiation.cc:407] Connection header received
0620 15:21:01.415683 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:01.415692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:01.415826 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:01.415919 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:01.417342 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.418850 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.425330 (+ 6480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.425933 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.426015 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.431919 (+ 5904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:01.431929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:01.431943 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:01.432719 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:01.432781 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:01.432817 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:01.432926 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:01.433210 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:21:01.433318 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:21:01.433928 18572 ts_tablet_manager.cc:1918] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:01.434103 18572 log.cc:1196] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/26d2da62842d46d49a0231d29ccb09ae
I20260620 15:21:01.434273 18573 ts_tablet_manager.cc:1939] T 26d2da62842d46d49a0231d29ccb09ae P 3c0835b1f1a9418eb7c4322befa9ebde: Deleting consensus metadata
I20260620 15:21:01.434605 18572 ts_tablet_manager.cc:1939] T 26d2da62842d46d49a0231d29ccb09ae P 428dcc5974634c32afc861f32236cf69: Deleting consensus metadata
I20260620 15:21:01.435171 18264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:01.436175 18067 catalog_manager.cc:4841] TS 428dcc5974634c32afc861f32236cf69 (127.0.97.67:42497): tablet 26d2da62842d46d49a0231d29ccb09ae (table test-table [id=bd474f81c0084ae9ab7eba16c64058be]) successfully deleted
I20260620 15:21:01.572324 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 18142
I20260620 15:21:01.572322 18068 catalog_manager.cc:4841] TS 3c0835b1f1a9418eb7c4322befa9ebde (127.0.97.65:40863): tablet 26d2da62842d46d49a0231d29ccb09ae (table test-table [id=bd474f81c0084ae9ab7eba16c64058be]) successfully deleted
I20260620 15:21:01.589730 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 18273
I20260620 15:21:01.596719 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:01.590526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.0.97.66:44549)
0620 15:21:01.590695 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:01.590702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:01.590776 (+ 74us) server_negotiation.cc:407] Connection header received
0620 15:21:01.590977 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:01.590982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:01.591062 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:01.591168 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:01.591569 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.592258 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.593595 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.594000 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.594130 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.595253 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:01.595262 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:01.595276 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:01.595844 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:01.595889 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:01.595908 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:01.596032 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:01.596135 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 15:21:01.596243 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:21:01.597306 18395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:01.685631 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 18273
I20260620 15:21:01.703109 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 18403
I20260620 15:21:01.711237 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:01.703891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.0.97.67:42497)
0620 15:21:01.704075 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:01.704081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:01.704103 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:21:01.704253 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:01.704259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:01.704367 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:01.704521 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:01.704924 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.706161 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.707646 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:01.708282 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:01.708412 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:01.709488 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:01.709498 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:01.709515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:01.710386 (+ 871us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:01.710458 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:01.710483 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:01.710607 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:01.710713 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 15:21:01.710813 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:21:01.711984 18525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:01.829471 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 18403
I20260620 15:21:01.844935 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 18052
I20260620 15:21:01.845984 18113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:01.952876 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 18052
2026-06-20T15:21:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2964 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:02 dist-test-slave-7s64 krb5kdc[18581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:02 dist-test-slave-7s64 krb5kdc[18581](info): set up 2 sockets
Jun 20 15:21:02 dist-test-slave-7s64 krb5kdc[18581](info): commencing operation
krb5kdc: starting...
W20260620 15:21:03.054088 389 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:03 dist-test-slave-7s64 krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:21:03.273595 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42253
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:03.588399 18594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:03.588689 18594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:03.588864 18594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:03.595858 18594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:03.595948 18594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:03.596029 18594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:03.596117 18594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:03.596196 18594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:03.605715 18594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:03.606961 18594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:03.608242 18594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:03.619603 18600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:03.619644 18602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:03.619837 18599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:03.620990 18594 server_base.cc:1034] running on GCE node
I20260620 15:21:03.621678 18594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:03.622886 18594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:03.624099 18594 hybrid_clock.cc:648] HybridClock initialized: now 1781968863624038 us; error 39 us; skew 500 ppm
I20260620 15:21:03.624470 18594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:03 dist-test-slave-7s64 krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:03.627485 18594 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:21:03.628894 18594 webserver.cc:460] Webserver started at http://127.0.97.126:43687/ using document root <none> and password file <none>
I20260620 15:21:03.629526 18594 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:03.629658 18594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:03.629967 18594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:03.632864 18594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "8a1a196e31ed42258f4f9d95d1e08737"
format_stamp: "Formatted at 2026-06-20 15:21:03 on dist-test-slave-7s64"
server_key: "b81f61dbd49a4a34a52ee2b61e2858aa"
server_key_iv: "6fd1c780872d1451ae5f5ca0ff9531b8"
server_key_version: "encryptionkey@0"
I20260620 15:21:03.633582 18594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "8a1a196e31ed42258f4f9d95d1e08737"
format_stamp: "Formatted at 2026-06-20 15:21:03 on dist-test-slave-7s64"
server_key: "b81f61dbd49a4a34a52ee2b61e2858aa"
server_key_iv: "6fd1c780872d1451ae5f5ca0ff9531b8"
server_key_version: "encryptionkey@0"
I20260620 15:21:03.638305 18594 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:21:03.641502 18609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:03.642699 18594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:21:03.642875 18594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "8a1a196e31ed42258f4f9d95d1e08737"
format_stamp: "Formatted at 2026-06-20 15:21:03 on dist-test-slave-7s64"
server_key: "b81f61dbd49a4a34a52ee2b61e2858aa"
server_key_iv: "6fd1c780872d1451ae5f5ca0ff9531b8"
server_key_version: "encryptionkey@0"
I20260620 15:21:03.643069 18594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:03.658380 18594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:03.662189 18594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:03.662458 18594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:03.680372 18594 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42253
I20260620 15:21:03.680408 18660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42253 every 8 connection(s)
I20260620 15:21:03.681681 18594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:03.685205 18661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:03.691767 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 18594
I20260620 15:21:03.692052 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:03.692575 389 external_mini_cluster.cc:1428] Setting key 92354bf1feb0601e8f04c89c34027280
I20260620 15:21:03.695158 18661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737: Bootstrap starting.
I20260620 15:21:03.698395 18661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:03.699602 18661 log.cc:824] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:03.702176 18661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737: No bootstrap required, opened a new log
Jun 20 15:21:03 dist-test-slave-7s64 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:03.710386 18661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1a196e31ed42258f4f9d95d1e08737" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42253 } }
I20260620 15:21:03.710726 18661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:03.710846 18661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a1a196e31ed42258f4f9d95d1e08737, State: Initialized, Role: FOLLOWER
I20260620 15:21:03.711460 18661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1a196e31ed42258f4f9d95d1e08737" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42253 } }
I20260620 15:21:03.711688 18661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:03.711824 18661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:03.712020 18661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:03.714488 18661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1a196e31ed42258f4f9d95d1e08737" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42253 } }
I20260620 15:21:03.714571 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:03.694235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.0.97.126:42253)
0620 15:21:03.694964 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:03.694975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:03.695012 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:21:03.696852 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:03.696873 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:03.697272 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:03.697563 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:03.699313 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:03.700721 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:03.702003 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:03.702695 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:03.702881 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:03.707215 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:03.707223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:03.707236 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:03.707275 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:03.710333 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:03.711159 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:03.711166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:03.711172 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:03.711326 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:03.711789 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:03.711795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:03.711799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:03.712468 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:03.712783 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:03.713235 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 15:21:03.713669 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":352,"threads_started":1}
I20260620 15:21:03.715044 18661 leader_election.cc:302] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a1a196e31ed42258f4f9d95d1e08737; no voters:
I20260620 15:21:03.715698 18661 leader_election.cc:288] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:03.715832 18666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:03.717432 18666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [term 1 LEADER]: Becoming Leader. State: Replica: 8a1a196e31ed42258f4f9d95d1e08737, State: Running, Role: LEADER
I20260620 15:21:03.718073 18661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:03.718012 18666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1a196e31ed42258f4f9d95d1e08737" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42253 } }
I20260620 15:21:03.721073 18668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a1a196e31ed42258f4f9d95d1e08737. Latest consensus state: current_term: 1 leader_uuid: "8a1a196e31ed42258f4f9d95d1e08737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1a196e31ed42258f4f9d95d1e08737" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42253 } } }
I20260620 15:21:03.721172 18667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a1a196e31ed42258f4f9d95d1e08737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a1a196e31ed42258f4f9d95d1e08737" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42253 } } }
I20260620 15:21:03.721462 18668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:03.721626 18667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:03.725057 18672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:03.730029 18672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:03.739900 18672 catalog_manager.cc:1347] Generated new cluster ID: 623ef85cc26344e7987193974db2d5bf
I20260620 15:21:03.740003 18672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:03.798339 18672 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:03.800191 18672 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:03.808902 18672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737: Generated new TSK 0
I20260620 15:21:03.809902 18672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:21:03.867218 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42253
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:04.183243 18687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:04.183482 18687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:04.183634 18687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:04.190608 18687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:04.190696 18687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:04.190903 18687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:21:04.200070 18687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:04.201313 18687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:04.202522 18687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:04.215531 18692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:04.215497 18693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:04.215586 18695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:04.215778 18687 server_base.cc:1034] running on GCE node
I20260620 15:21:04.216581 18687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:04.217361 18687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:04.218638 18687 hybrid_clock.cc:648] HybridClock initialized: now 1781968864218575 us; error 39 us; skew 500 ppm
I20260620 15:21:04.218940 18687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:04 dist-test-slave-7s64 krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:04.222110 18687 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:21:04.223557 18687 webserver.cc:460] Webserver started at http://127.0.97.65:44115/ using document root <none> and password file <none>
I20260620 15:21:04.224119 18687 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:04.224216 18687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:04.224452 18687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:04.227068 18687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "1c913262e3ad4a59bcca57105c7bb768"
format_stamp: "Formatted at 2026-06-20 15:21:04 on dist-test-slave-7s64"
server_key: "b1b3f451dc281a44e3d011941c564dc6"
server_key_iv: "79e083d60a3f209c8d7ae010dea0502d"
server_key_version: "encryptionkey@0"
I20260620 15:21:04.227739 18687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "1c913262e3ad4a59bcca57105c7bb768"
format_stamp: "Formatted at 2026-06-20 15:21:04 on dist-test-slave-7s64"
server_key: "b1b3f451dc281a44e3d011941c564dc6"
server_key_iv: "79e083d60a3f209c8d7ae010dea0502d"
server_key_version: "encryptionkey@0"
I20260620 15:21:04.232468 18687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 15:21:04.235553 18702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:04.236784 18687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260620 15:21:04.236902 18687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "1c913262e3ad4a59bcca57105c7bb768"
format_stamp: "Formatted at 2026-06-20 15:21:04 on dist-test-slave-7s64"
server_key: "b1b3f451dc281a44e3d011941c564dc6"
server_key_iv: "79e083d60a3f209c8d7ae010dea0502d"
server_key_version: "encryptionkey@0"
I20260620 15:21:04.237046 18687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:04.261444 18687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:04.265342 18687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:04.265550 18687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:04.266309 18687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:04.267545 18687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:04.267613 18687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:04.267689 18687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:04.267732 18687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:04.300251 18687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:45405
I20260620 15:21:04.300293 18814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:45405 every 8 connection(s)
I20260620 15:21:04.301533 18687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:21:04.309650 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 18687
I20260620 15:21:04.309867 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:21:04.310328 389 external_mini_cluster.cc:1428] Setting key 9b99de7bf602306ec9fa3bbe367c67ec
Jun 20 15:21:04 dist-test-slave-7s64 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:04.322569 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:04.304256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:33033 (local address 127.0.97.126:42253)
0620 15:21:04.304755 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:04.304759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:04.305725 (+ 966us) server_negotiation.cc:407] Connection header received
0620 15:21:04.307487 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:04.307493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:04.307582 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:04.307713 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:04.309239 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:04.310060 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:04.311897 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:04.312488 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:04.312640 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:04.315874 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:04.315883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:04.315890 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:04.315946 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:04.319576 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:04.320235 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:04.320244 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:04.320250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:04.320420 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:04.320928 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:04.320931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:04.320933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:04.321175 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:04.321284 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:04.322158 (+ 874us) server_negotiation.cc:299] Negotiation successful
0620 15:21:04.322319 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":284,"threads_started":1}
I20260620 15:21:04.323865 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:04.304851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42253 (local address 127.0.97.65:33033)
0620 15:21:04.305545 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:04.305591 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:04.307229 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:04.307948 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:04.307956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:04.308441 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:04.309075 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:04.309104 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:04.310212 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:04.310216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:04.311733 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:04.311753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:04.312691 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:04.312697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:04.312882 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:04.313560 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:04.313595 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:04.315666 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:04.319706 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:04.319711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:04.319722 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:04.320116 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:04.320543 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:04.320551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:04.320556 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:04.320809 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:04.321304 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:04.321313 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:04.321735 (+ 422us) client_negotiation.cc:769] Sending connection context
0620 15:21:04.322009 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 15:21:04.322523 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":296,"threads_started":1}
I20260620 15:21:04.325325 18815 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42253
I20260620 15:21:04.325609 18815 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:04.326309 18815 heartbeater.cc:511] Master 127.0.97.126:42253 requested a full tablet report, sending...
I20260620 15:21:04.328277 18626 ts_manager.cc:194] Registered new tserver with Master: 1c913262e3ad4a59bcca57105c7bb768 (127.0.97.65:45405)
I20260620 15:21:04.330030 18626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:33033
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:21:04.361348 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42253
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:04.675273 18821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:04.675514 18821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:04.675602 18821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:04.682550 18821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:04.682641 18821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:04.682808 18821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:21:04.691937 18821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:04.693064 18821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:04.694245 18821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:04.706630 18826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:04.706661 18827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:04.706823 18829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:04.707338 18821 server_base.cc:1034] running on GCE node
I20260620 15:21:04.707868 18821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:04.708801 18821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:04.710016 18821 hybrid_clock.cc:648] HybridClock initialized: now 1781968864709976 us; error 54 us; skew 500 ppm
I20260620 15:21:04.710314 18821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:04 dist-test-slave-7s64 krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:04.713339 18821 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:21:04.714661 18821 webserver.cc:460] Webserver started at http://127.0.97.66:36675/ using document root <none> and password file <none>
I20260620 15:21:04.715253 18821 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:04.715345 18821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:04.715579 18821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:04.718338 18821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "a4ce764e24ea41a4ba406f5bf274db3a"
format_stamp: "Formatted at 2026-06-20 15:21:04 on dist-test-slave-7s64"
server_key: "8ce24dde98d48ef362ed6667cd5f6443"
server_key_iv: "2a8a4d1e34a0883c689eacceca532edd"
server_key_version: "encryptionkey@0"
I20260620 15:21:04.719092 18821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "a4ce764e24ea41a4ba406f5bf274db3a"
format_stamp: "Formatted at 2026-06-20 15:21:04 on dist-test-slave-7s64"
server_key: "8ce24dde98d48ef362ed6667cd5f6443"
server_key_iv: "2a8a4d1e34a0883c689eacceca532edd"
server_key_version: "encryptionkey@0"
I20260620 15:21:04.723989 18821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:21:04.727278 18836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:04.728597 18821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:21:04.728868 18821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "a4ce764e24ea41a4ba406f5bf274db3a"
format_stamp: "Formatted at 2026-06-20 15:21:04 on dist-test-slave-7s64"
server_key: "8ce24dde98d48ef362ed6667cd5f6443"
server_key_iv: "2a8a4d1e34a0883c689eacceca532edd"
server_key_version: "encryptionkey@0"
I20260620 15:21:04.729095 18821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:04.744309 18821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:04.748378 18821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:04.748701 18821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:04.749557 18821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:04.750718 18821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:04.750815 18821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:04.750967 18821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:04.751042 18821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:04.782563 18821 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:35429
I20260620 15:21:04.782635 18948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:35429 every 8 connection(s)
I20260620 15:21:04.783723 18821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:21:04.790535 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 18821
I20260620 15:21:04.790722 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:21:04.791174 389 external_mini_cluster.cc:1428] Setting key a6c867f4b2fea4d948c74c4de7754e69
Jun 20 15:21:04 dist-test-slave-7s64 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:04.803838 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:04.786107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41535 (local address 127.0.97.126:42253)
0620 15:21:04.786280 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:04.786285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:04.787305 (+ 1020us) server_negotiation.cc:407] Connection header received
0620 15:21:04.789141 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:04.789147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:04.789236 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:04.789344 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:04.790718 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:04.791524 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:04.793316 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:04.793722 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:04.793852 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:04.797590 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:04.797598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:04.797606 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:04.797646 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:04.800917 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:04.801586 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:04.801599 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:04.801601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:04.801730 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:04.802207 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:04.802211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:04.802213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:04.802480 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:04.802595 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:04.803379 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 15:21:04.803542 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:21:04.805315 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:04.786492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42253 (local address 127.0.97.66:41535)
0620 15:21:04.787144 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:04.787196 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:04.788891 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:04.789556 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:04.789564 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:04.790020 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:04.790549 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:04.790564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:04.791674 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:04.791678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:04.793180 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:04.793191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:04.793929 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:04.793936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:04.794150 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:04.794798 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:04.794818 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:04.797357 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:04.801050 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:04.801058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:04.801072 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:04.801462 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:04.801837 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:04.801845 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:04.801851 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:04.802075 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:04.802604 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:04.802614 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:04.803011 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 15:21:04.803288 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 15:21:04.803824 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":264,"threads_started":1}
I20260620 15:21:04.806664 18949 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42253
I20260620 15:21:04.806936 18949 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:04.807593 18949 heartbeater.cc:511] Master 127.0.97.126:42253 requested a full tablet report, sending...
I20260620 15:21:04.809021 18626 ts_manager.cc:194] Registered new tserver with Master: a4ce764e24ea41a4ba406f5bf274db3a (127.0.97.66:35429)
I20260620 15:21:04.809831 18626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:41535
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:21:04.841764 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42253
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:05.152530 18954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:05.152798 18954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:05.152891 18954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:05.159755 18954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:05.159850 18954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:05.159960 18954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:21:05.169126 18954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:05.170272 18954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:05.171447 18954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:05.183842 18959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:05.183796 18962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:05.183882 18960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:05.185112 18954 server_base.cc:1034] running on GCE node
I20260620 15:21:05.185645 18954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:05.186460 18954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:05.187693 18954 hybrid_clock.cc:648] HybridClock initialized: now 1781968865187647 us; error 48 us; skew 500 ppm
I20260620 15:21:05.187980 18954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:05 dist-test-slave-7s64 krb5kdc[18581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:05.196352 18954 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:21:05.197924 18954 webserver.cc:460] Webserver started at http://127.0.97.67:45967/ using document root <none> and password file <none>
I20260620 15:21:05.198604 18954 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:05.198755 18954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:05.199067 18954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:05.202098 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "c5a967eca9d041a8bb61da5e32a541f1"
format_stamp: "Formatted at 2026-06-20 15:21:05 on dist-test-slave-7s64"
server_key: "4051764619059d3ae0af77871495bd26"
server_key_iv: "e94be109b241426aeb64babb530344a9"
server_key_version: "encryptionkey@0"
I20260620 15:21:05.202885 18954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "c5a967eca9d041a8bb61da5e32a541f1"
format_stamp: "Formatted at 2026-06-20 15:21:05 on dist-test-slave-7s64"
server_key: "4051764619059d3ae0af77871495bd26"
server_key_iv: "e94be109b241426aeb64babb530344a9"
server_key_version: "encryptionkey@0"
I20260620 15:21:05.207567 18954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 15:21:05.210778 18969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:05.211992 18954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:21:05.212141 18954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "c5a967eca9d041a8bb61da5e32a541f1"
format_stamp: "Formatted at 2026-06-20 15:21:05 on dist-test-slave-7s64"
server_key: "4051764619059d3ae0af77871495bd26"
server_key_iv: "e94be109b241426aeb64babb530344a9"
server_key_version: "encryptionkey@0"
I20260620 15:21:05.212332 18954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:05.244324 18954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:05.248584 18954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:05.248827 18954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:05.249696 18954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:05.250942 18954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:05.251015 18954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:05.251107 18954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:05.251160 18954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:05.283195 18954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:36713
I20260620 15:21:05.283232 19081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:36713 every 8 connection(s)
I20260620 15:21:05.284420 18954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:21:05.291280 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 18954
I20260620 15:21:05.291467 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:21:05.292014 389 external_mini_cluster.cc:1428] Setting key 6a7b5c6c332fb710ca855dad3ebf970c
Jun 20 15:21:05 dist-test-slave-7s64 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:05.304813 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.286986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:42595 (local address 127.0.97.126:42253)
0620 15:21:05.287220 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.287226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.288362 (+ 1136us) server_negotiation.cc:407] Connection header received
0620 15:21:05.290154 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.290158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.290251 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.290349 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:05.292143 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.293098 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.294831 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.295278 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.295373 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.298724 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:05.298729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:05.298732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:05.298764 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:05.301980 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:05.302612 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:05.302617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:05.302620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:05.302763 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:05.303139 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:05.303151 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:05.303155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:05.303502 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:05.303643 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.304244 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.304449 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:21:05.305635 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.287525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42253 (local address 127.0.97.67:42595)
0620 15:21:05.288206 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:05.288249 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:05.289904 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:05.290543 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:05.290551 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:05.291179 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:05.291936 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.291959 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.293258 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.293264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.294676 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.294691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.295461 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.295466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.295643 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.296284 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:05.296304 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:05.298522 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:05.302103 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:05.302108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:05.302119 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:05.302495 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:05.302872 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:05.302875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:05.302877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:05.303033 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:05.303630 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:05.303635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:05.303909 (+ 274us) client_negotiation.cc:769] Sending connection context
0620 15:21:05.304117 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 15:21:05.304446 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":286,"threads_started":1}
I20260620 15:21:05.307063 19082 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42253
I20260620 15:21:05.307358 19082 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:05.308022 19082 heartbeater.cc:511] Master 127.0.97.126:42253 requested a full tablet report, sending...
I20260620 15:21:05.309346 18626 ts_manager.cc:194] Registered new tserver with Master: c5a967eca9d041a8bb61da5e32a541f1 (127.0.97.67:36713)
I20260620 15:21:05.310158 18626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:42595
I20260620 15:21:05.319029 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:21:05.332932 18815 heartbeater.cc:503] Master 127.0.97.126:42253 was elected leader, sending a full tablet report...
I20260620 15:21:05.335592 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.322867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.0.97.126:42253)
0620 15:21:05.323062 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.323070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.323447 (+ 377us) server_negotiation.cc:407] Connection header received
0620 15:21:05.323557 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.323561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.323614 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.323693 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:05.325026 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.325716 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.326956 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.327331 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.327416 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.329878 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:05.329882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:05.329885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:05.329913 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:05.332522 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:05.333520 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:05.333525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:05.333529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:05.333751 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:05.334247 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:05.334251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:05.334256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:05.334575 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:05.334779 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.335106 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.335291 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:21:05.338824 18626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:21:05.345507 18626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:21:05.368885 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.357536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:35429 (local address 127.0.0.1:57586)
0620 15:21:05.360265 (+ 2729us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:05.360284 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:05.360548 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:05.361430 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:05.361435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:05.361456 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:05.362087 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.362107 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.364009 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.364016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.366141 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.366156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.367842 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.367848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.368043 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.368100 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 15:21:05.368231 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 15:21:05.368415 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2589,"mutex_wait_us":220,"thread_start_us":786,"threads_started":1}
I20260620 15:21:05.368916 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.358146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.0.97.66:35429)
0620 15:21:05.359922 (+ 1776us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.359934 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.360396 (+ 462us) server_negotiation.cc:407] Connection header received
0620 15:21:05.360774 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.360782 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.360959 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.361123 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:05.362290 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.363842 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.366306 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.367641 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.367785 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.367982 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.368308 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.368481 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1612,"thread_start_us":1476,"threads_started":1}
I20260620 15:21:05.369877 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.357574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.0.97.65:45405)
0620 15:21:05.360464 (+ 2890us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.360473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.360523 (+ 50us) server_negotiation.cc:407] Connection header received
0620 15:21:05.360629 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.360639 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.360895 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.361786 (+ 891us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:05.362041 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.363690 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.367364 (+ 3674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.368834 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.369138 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.369293 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.369459 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.369626 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2690,"thread_start_us":2541,"threads_started":1}
I20260620 15:21:05.370002 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.357206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:45405 (local address 127.0.0.1:59738)
0620 15:21:05.358472 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:05.358510 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:05.358684 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:05.361164 (+ 2480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:05.361173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:05.361209 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:05.361815 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.361830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.363864 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.363871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.367177 (+ 3306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.367189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.369064 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.369071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.369293 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.369334 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:21:05.369433 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:21:05.369565 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":200,"thread_start_us":3823,"threads_started":1}
I20260620 15:21:05.371057 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.358585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.0.97.67:36713)
0620 15:21:05.359434 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.359448 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.360855 (+ 1407us) server_negotiation.cc:407] Connection header received
0620 15:21:05.361153 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.361161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.361346 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.361465 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:05.362472 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.363800 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.366877 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.368272 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.368405 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.368652 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.370501 (+ 1849us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.370661 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":400,"threads_started":1}
I20260620 15:21:05.371814 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.358006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:36713 (local address 127.0.0.1:49676)
0620 15:21:05.360758 (+ 2752us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:05.360776 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:05.360955 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:05.361698 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:05.361701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:05.361722 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:05.362308 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.362318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.363969 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.363974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.366698 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.366723 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.368807 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.368814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.369049 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.370315 (+ 1266us) client_negotiation.cc:769] Sending connection context
0620 15:21:05.371255 (+ 940us) client_negotiation.cc:240] Negotiation successful
0620 15:21:05.371381 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2614,"mutex_wait_us":184,"thread_start_us":1949,"threads_started":1}
I20260620 15:21:05.373382 18750 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4f84ce2e2c45ad821be156723be6f5 (DEFAULT_TABLE table=test-table [id=1e4bd947c5c64c3c9b9b09ceb2dec0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:05.373382 18884 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4f84ce2e2c45ad821be156723be6f5 (DEFAULT_TABLE table=test-table [id=1e4bd947c5c64c3c9b9b09ceb2dec0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:05.374825 18750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4f84ce2e2c45ad821be156723be6f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:05.374845 18884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4f84ce2e2c45ad821be156723be6f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:05.374809 19017 tablet_service.cc:1467] Processing CreateTablet for tablet 6e4f84ce2e2c45ad821be156723be6f5 (DEFAULT_TABLE table=test-table [id=1e4bd947c5c64c3c9b9b09ceb2dec0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:05.376163 19017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e4f84ce2e2c45ad821be156723be6f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:05.387475 19100 tablet_bootstrap.cc:492] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: Bootstrap starting.
I20260620 15:21:05.389186 19102 tablet_bootstrap.cc:492] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: Bootstrap starting.
I20260620 15:21:05.392953 19102 tablet_bootstrap.cc:654] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:05.392956 19100 tablet_bootstrap.cc:654] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:05.394264 19100 log.cc:824] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:05.394268 19102 log.cc:824] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:05.395073 19101 tablet_bootstrap.cc:492] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: Bootstrap starting.
I20260620 15:21:05.398507 19101 tablet_bootstrap.cc:654] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:05.399194 19102 tablet_bootstrap.cc:492] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: No bootstrap required, opened a new log
I20260620 15:21:05.399469 19102 ts_tablet_manager.cc:1397] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260620 15:21:05.399740 19101 log.cc:824] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:05.399828 19100 tablet_bootstrap.cc:492] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: No bootstrap required, opened a new log
I20260620 15:21:05.400055 19100 ts_tablet_manager.cc:1397] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260620 15:21:05.402629 19101 tablet_bootstrap.cc:492] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: No bootstrap required, opened a new log
I20260620 15:21:05.402872 19101 ts_tablet_manager.cc:1397] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:21:05.407004 19102 raft_consensus.cc:348] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 } } peers { permanent_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35429 } } peers { permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 } }
I20260620 15:21:05.407353 19102 raft_consensus.cc:374] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:05.407483 19102 raft_consensus.cc:729] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c913262e3ad4a59bcca57105c7bb768, State: Initialized, Role: FOLLOWER
I20260620 15:21:05.408082 19102 consensus_queue.cc:260] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 } } peers { permanent_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35429 } } peers { permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 } }
I20260620 15:21:05.410285 19100 raft_consensus.cc:348] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 } } peers { permanent_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35429 } } peers { permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 } }
I20260620 15:21:05.410669 19100 raft_consensus.cc:374] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:05.410766 19100 raft_consensus.cc:729] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a967eca9d041a8bb61da5e32a541f1, State: Initialized, Role: FOLLOWER
I20260620 15:21:05.411279 19100 consensus_queue.cc:260] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 } } peers { permanent_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35429 } } peers { permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 } }
I20260620 15:21:05.411501 19102 ts_tablet_manager.cc:1428] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: Time spent starting tablet: real 0.012s user 0.003s sys 0.011s
I20260620 15:21:05.412730 19082 heartbeater.cc:503] Master 127.0.97.126:42253 was elected leader, sending a full tablet report...
I20260620 15:21:05.413318 19100 ts_tablet_manager.cc:1428] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260620 15:21:05.413165 19101 raft_consensus.cc:348] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 } } peers { permanent_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35429 } } peers { permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 } }
I20260620 15:21:05.413569 19101 raft_consensus.cc:374] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:05.413641 19101 raft_consensus.cc:729] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ce764e24ea41a4ba406f5bf274db3a, State: Initialized, Role: FOLLOWER
I20260620 15:21:05.414198 19101 consensus_queue.cc:260] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 } } peers { permanent_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35429 } } peers { permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 } }
I20260620 15:21:05.415277 18949 heartbeater.cc:503] Master 127.0.97.126:42253 was elected leader, sending a full tablet report...
I20260620 15:21:05.416029 19101 ts_tablet_manager.cc:1428] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: Time spent starting tablet: real 0.013s user 0.004s sys 0.010s
I20260620 15:21:05.450107 19108 raft_consensus.cc:482] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:21:05.450402 19108 raft_consensus.cc:504] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 } } peers { permanent_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35429 } } peers { permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 } }
I20260620 15:21:05.451901 19108 leader_election.cc:288] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c913262e3ad4a59bcca57105c7bb768 (127.0.97.65:45405), c5a967eca9d041a8bb61da5e32a541f1 (127.0.97.67:36713)
I20260620 15:21:05.458544 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.452292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:43343 (local address 127.0.97.67:36713)
0620 15:21:05.452676 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.452685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.452991 (+ 306us) server_negotiation.cc:407] Connection header received
0620 15:21:05.453239 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.453243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.453297 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.453393 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:05.453942 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.454639 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.456575 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.457752 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.457883 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.457984 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.458277 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.458370 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260620 15:21:05.458949 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.452404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39863 (local address 127.0.97.65:45405)
0620 15:21:05.452628 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.452635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.453288 (+ 653us) server_negotiation.cc:407] Connection header received
0620 15:21:05.453480 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.453484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.453537 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.453633 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:05.454135 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.454814 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.456780 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.458006 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.458117 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.458208 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.458646 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.458787 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:21:05.459043 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.452133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:36713 (local address 127.0.97.66:43343)
0620 15:21:05.452882 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:05.452897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:05.453071 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:05.453408 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:05.453413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:05.453438 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:05.453787 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.453796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.454767 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.454774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.456399 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.456411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.457943 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.457947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.458064 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.458118 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 15:21:05.458229 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 15:21:05.458384 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":23,"thread_start_us":480,"threads_started":1}
I20260620 15:21:05.459055 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.452144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:45405 (local address 127.0.97.66:39863)
0620 15:21:05.453191 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:05.453204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:05.453349 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:05.453648 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:05.453652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:05.453674 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:05.453996 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.454004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.454950 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.454955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.456350 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:05.456361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.458227 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.458233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:05.458454 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.458498 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 15:21:05.458588 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 15:21:05.458718 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":92,"thread_start_us":666,"threads_started":1}
I20260620 15:21:05.459913 19037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4f84ce2e2c45ad821be156723be6f5" candidate_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a967eca9d041a8bb61da5e32a541f1" is_pre_election: true
I20260620 15:21:05.459940 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4f84ce2e2c45ad821be156723be6f5" candidate_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c913262e3ad4a59bcca57105c7bb768" is_pre_election: true
I20260620 15:21:05.460472 19037 raft_consensus.cc:2454] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4ce764e24ea41a4ba406f5bf274db3a in term 0.
I20260620 15:21:05.460546 18770 raft_consensus.cc:2454] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4ce764e24ea41a4ba406f5bf274db3a in term 0.
I20260620 15:21:05.461243 18840 leader_election.cc:302] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c913262e3ad4a59bcca57105c7bb768, a4ce764e24ea41a4ba406f5bf274db3a, c5a967eca9d041a8bb61da5e32a541f1; no voters:
I20260620 15:21:05.461648 19108 raft_consensus.cc:2781] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:21:05.461797 19108 raft_consensus.cc:482] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:21:05.461902 19108 raft_consensus.cc:3037] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:05.464310 19108 raft_consensus.cc:504] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 } } peers { permanent_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35429 } } peers { permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 } }
I20260620 15:21:05.464881 19108 leader_election.cc:288] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [CANDIDATE]: Term 1 election: Requested vote from peers 1c913262e3ad4a59bcca57105c7bb768 (127.0.97.65:45405), c5a967eca9d041a8bb61da5e32a541f1 (127.0.97.67:36713)
I20260620 15:21:05.465286 18770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4f84ce2e2c45ad821be156723be6f5" candidate_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c913262e3ad4a59bcca57105c7bb768"
I20260620 15:21:05.465318 19037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e4f84ce2e2c45ad821be156723be6f5" candidate_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a967eca9d041a8bb61da5e32a541f1"
I20260620 15:21:05.465488 18770 raft_consensus.cc:3037] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:05.465504 19037 raft_consensus.cc:3037] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:05.467547 18770 raft_consensus.cc:2454] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4ce764e24ea41a4ba406f5bf274db3a in term 1.
I20260620 15:21:05.467612 19037 raft_consensus.cc:2454] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4ce764e24ea41a4ba406f5bf274db3a in term 1.
I20260620 15:21:05.467947 18838 leader_election.cc:302] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c913262e3ad4a59bcca57105c7bb768, a4ce764e24ea41a4ba406f5bf274db3a; no voters:
I20260620 15:21:05.468246 19108 raft_consensus.cc:2781] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:05.468700 19108 raft_consensus.cc:686] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 1 LEADER]: Becoming Leader. State: Replica: a4ce764e24ea41a4ba406f5bf274db3a, State: Running, Role: LEADER
I20260620 15:21:05.469142 19108 consensus_queue.cc:237] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 } } peers { permanent_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35429 } } peers { permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 } }
I20260620 15:21:05.473569 18624 catalog_manager.cc:5495] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a reported cstate change: term changed from 0 to 1, leader changed from <none> to a4ce764e24ea41a4ba406f5bf274db3a (127.0.97.66). New cstate: current_term: 1 leader_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4ce764e24ea41a4ba406f5bf274db3a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 35429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 } health_report { overall_health: UNKNOWN } } }
I20260620 15:21:05.492472 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.487169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.0.97.66:35429)
0620 15:21:05.487380 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.487386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.487495 (+ 109us) server_negotiation.cc:407] Connection header received
0620 15:21:05.487712 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.487717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.487785 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.487929 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:05.488308 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.489051 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.490388 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.490777 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.490902 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.491487 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:05.491912 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:05.492099 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.492220 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.492263 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:21:05.499879 19037 raft_consensus.cc:1261] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1 [term 1 FOLLOWER]: Refusing update from remote peer a4ce764e24ea41a4ba406f5bf274db3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:05.500234 18770 raft_consensus.cc:1261] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768 [term 1 FOLLOWER]: Refusing update from remote peer a4ce764e24ea41a4ba406f5bf274db3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:05.500844 19111 consensus_queue.cc:1035] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a967eca9d041a8bb61da5e32a541f1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:05.501273 19108 consensus_queue.cc:1035] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c913262e3ad4a59bcca57105c7bb768" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:05.515977 19115 mvcc.cc:205] Tried to move back new op lower bound from 7298944473078939648 to 7298944472965238784. Current Snapshot: MvccSnapshot[applied={T|T < 7298944473078939648}]
I20260620 15:21:05.527066 18626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:21:05.527326 18626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:21:05.530454 18626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a1a196e31ed42258f4f9d95d1e08737: Sending DeleteTablet for 3 replicas of tablet 6e4f84ce2e2c45ad821be156723be6f5
I20260620 15:21:05.531528 18750 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4f84ce2e2c45ad821be156723be6f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:05 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:59738
I20260620 15:21:05.531535 18884 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4f84ce2e2c45ad821be156723be6f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:05 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:57586
I20260620 15:21:05.531528 19017 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e4f84ce2e2c45ad821be156723be6f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:05 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49676
I20260620 15:21:05.537655 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 18687
I20260620 15:21:05.545495 19125 tablet_replica.cc:330] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: stopping tablet replica
I20260620 15:21:05.545828 19125 raft_consensus.cc:2229] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:21:05.546305 19125 raft_consensus.cc:2258] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:05.546937 19126 tablet_replica.cc:330] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: stopping tablet replica
I20260620 15:21:05.547317 19126 raft_consensus.cc:2229] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:21:05.547705 19126 raft_consensus.cc:2258] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:05.548576 19124 tablet_replica.cc:330] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: stopping tablet replica
I20260620 15:21:05.548919 19124 raft_consensus.cc:2229] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:21:05.549537 19124 raft_consensus.cc:2258] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:05.549965 19126 ts_tablet_manager.cc:1905] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:05.551280 19125 ts_tablet_manager.cc:1905] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:05.556820 19126 ts_tablet_manager.cc:1918] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:05.557065 19126 log.cc:1196] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/6e4f84ce2e2c45ad821be156723be6f5
I20260620 15:21:05.557588 19126 ts_tablet_manager.cc:1939] T 6e4f84ce2e2c45ad821be156723be6f5 P c5a967eca9d041a8bb61da5e32a541f1: Deleting consensus metadata
I20260620 15:21:05.558070 19125 ts_tablet_manager.cc:1918] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:05.558236 19125 log.cc:1196] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/6e4f84ce2e2c45ad821be156723be6f5
I20260620 15:21:05.558749 19125 ts_tablet_manager.cc:1939] T 6e4f84ce2e2c45ad821be156723be6f5 P 1c913262e3ad4a59bcca57105c7bb768: Deleting consensus metadata
I20260620 15:21:05.559579 18613 catalog_manager.cc:4841] TS c5a967eca9d041a8bb61da5e32a541f1 (127.0.97.67:36713): tablet 6e4f84ce2e2c45ad821be156723be6f5 (table test-table [id=1e4bd947c5c64c3c9b9b09ceb2dec0aa]) successfully deleted
I20260620 15:21:05.560379 18611 catalog_manager.cc:4841] TS 1c913262e3ad4a59bcca57105c7bb768 (127.0.97.65:45405): tablet 6e4f84ce2e2c45ad821be156723be6f5 (table test-table [id=1e4bd947c5c64c3c9b9b09ceb2dec0aa]) successfully deleted
I20260620 15:21:05.561424 19124 ts_tablet_manager.cc:1905] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:05.566311 19124 ts_tablet_manager.cc:1918] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:05.566505 19124 log.cc:1196] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/6e4f84ce2e2c45ad821be156723be6f5
I20260620 15:21:05.567049 19124 ts_tablet_manager.cc:1939] T 6e4f84ce2e2c45ad821be156723be6f5 P a4ce764e24ea41a4ba406f5bf274db3a: Deleting consensus metadata
I20260620 15:21:05.568699 18610 catalog_manager.cc:4841] TS a4ce764e24ea41a4ba406f5bf274db3a (127.0.97.66:35429): tablet 6e4f84ce2e2c45ad821be156723be6f5 (table test-table [id=1e4bd947c5c64c3c9b9b09ceb2dec0aa]) successfully deleted
Jun 20 15:21:05 dist-test-slave-7s64 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:21:05.577840 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.538879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.0.97.65:45405)
0620 15:21:05.539044 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.539050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.553229 (+ 14179us) server_negotiation.cc:407] Connection header received
0620 15:21:05.553644 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.553650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.553742 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.553813 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:05.555534 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.556725 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.558678 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.559367 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.559461 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.570493 (+ 11032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:05.570510 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:05.570528 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:05.570581 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:05.574774 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:05.575449 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:05.575459 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:05.575465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:05.575642 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:05.576007 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:05.576014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:05.576018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:05.576571 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:05.576944 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.577134 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.577372 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:21:05.578657 18810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:05.667894 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 18687
I20260620 15:21:05.685389 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 18821
Jun 20 15:21:05 dist-test-slave-7s64 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:21:05.701160 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.686219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.0.97.66:35429)
0620 15:21:05.686694 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.686698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.686712 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:21:05.686761 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.686765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.686827 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.686948 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:05.688238 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.689204 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.690491 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.690903 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.691036 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.695233 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:05.695244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:05.695258 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:05.695292 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:05.698381 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:05.699099 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:05.699106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:05.699108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:05.699256 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:05.699619 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:05.699625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:05.699628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:05.699973 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:05.700312 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.700567 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.700772 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260620 15:21:05.701881 18944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:05.813633 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 18821
I20260620 15:21:05.833654 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 18954
Jun 20 15:21:05 dist-test-slave-7s64 krb5kdc[18581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:21:05.848423 19099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:05.834421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.0.97.67:36713)
0620 15:21:05.834590 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:05.834595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:05.834613 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:21:05.834740 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:05.834744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:05.834811 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:05.834893 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:05.836170 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.836980 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.838201 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:05.838596 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:05.838702 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:05.842986 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:05.842998 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:05.843013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:05.843042 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:05.845782 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:05.846429 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:05.846435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:05.846437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:05.846602 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:05.847163 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:05.847166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:05.847168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:05.847490 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:05.847772 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:05.848007 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 15:21:05.848163 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:21:05.849128 19077 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:05.937354 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 18954
I20260620 15:21:05.953537 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 18594
I20260620 15:21:05.954667 18656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:06.064090 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 18594
2026-06-20T15:21:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4108 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T15:21:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:06Z Disabled control of system clock
I20260620 15:21:06.137023 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37123
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:21:06.448534 19135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:06.448791 19135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:06.448954 19135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:06.455976 19135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:06.456061 19135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:06.456139 19135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:06.456240 19135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:06.456326 19135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:06.465749 19135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37123
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:06.467177 19135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:06.468990 19135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:06.480556 19141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:06.480479 19143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:06.480969 19140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:06.481520 19135 server_base.cc:1034] running on GCE node
I20260620 15:21:06.482172 19135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:06.483351 19135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:06.484624 19135 hybrid_clock.cc:648] HybridClock initialized: now 1781968866484562 us; error 31 us; skew 500 ppm
I20260620 15:21:06.484972 19135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:06.487064 19135 webserver.cc:460] Webserver started at http://127.0.97.126:34977/ using document root <none> and password file <none>
I20260620 15:21:06.487684 19135 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:06.487797 19135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:06.488116 19135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:06.490841 19135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "fe59704451594e3eab64d78553356c38"
format_stamp: "Formatted at 2026-06-20 15:21:06 on dist-test-slave-7s64"
server_key: "1c5cb42fa8cdaca8c4ac69e1aebac6ce"
server_key_iv: "4978fbec7bf9ad186e5ab515744decfd"
server_key_version: "encryptionkey@0"
I20260620 15:21:06.491595 19135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "fe59704451594e3eab64d78553356c38"
format_stamp: "Formatted at 2026-06-20 15:21:06 on dist-test-slave-7s64"
server_key: "1c5cb42fa8cdaca8c4ac69e1aebac6ce"
server_key_iv: "4978fbec7bf9ad186e5ab515744decfd"
server_key_version: "encryptionkey@0"
I20260620 15:21:06.496321 19135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260620 15:21:06.499538 19149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:06.500871 19135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:21:06.501039 19135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "fe59704451594e3eab64d78553356c38"
format_stamp: "Formatted at 2026-06-20 15:21:06 on dist-test-slave-7s64"
server_key: "1c5cb42fa8cdaca8c4ac69e1aebac6ce"
server_key_iv: "4978fbec7bf9ad186e5ab515744decfd"
server_key_version: "encryptionkey@0"
I20260620 15:21:06.501232 19135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:06.532109 19135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:06.532847 19135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:06.533100 19135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:06.551064 19135 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37123
I20260620 15:21:06.551098 19200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37123 every 8 connection(s)
I20260620 15:21:06.552265 19135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:06.555284 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 19135
I20260620 15:21:06.555506 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:06.555701 19201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:06.555966 389 external_mini_cluster.cc:1428] Setting key 36769e0582e78682ee8643cb8490ece4
I20260620 15:21:06.568629 19201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38: Bootstrap starting.
I20260620 15:21:06.568915 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:06.557790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.0.97.126:37123)
0620 15:21:06.558773 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:06.558785 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:06.558826 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:21:06.560295 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:06.560313 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:06.560811 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:06.561113 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:06.561690 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:06.563003 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:06.564750 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:06.565381 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:06.565609 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:06.566738 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:06.566747 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:06.566760 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:06.567249 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:06.567317 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:06.567337 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:06.567454 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:06.567682 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 15:21:06.567961 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":376,"threads_started":1}
I20260620 15:21:06.571743 19201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:06.572759 19201 log.cc:824] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:06.575273 19201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38: No bootstrap required, opened a new log
I20260620 15:21:06.582317 19201 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe59704451594e3eab64d78553356c38" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } }
I20260620 15:21:06.582665 19201 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:06.582779 19201 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe59704451594e3eab64d78553356c38, State: Initialized, Role: FOLLOWER
I20260620 15:21:06.583343 19201 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe59704451594e3eab64d78553356c38" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } }
I20260620 15:21:06.583580 19201 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:06.583726 19201 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:06.583899 19201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:06.585991 19201 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe59704451594e3eab64d78553356c38" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } }
I20260620 15:21:06.586465 19201 leader_election.cc:302] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe59704451594e3eab64d78553356c38; no voters:
I20260620 15:21:06.587019 19201 leader_election.cc:288] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:06.587127 19206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:06.587823 19206 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [term 1 LEADER]: Becoming Leader. State: Replica: fe59704451594e3eab64d78553356c38, State: Running, Role: LEADER
I20260620 15:21:06.588403 19206 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe59704451594e3eab64d78553356c38" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } }
I20260620 15:21:06.589156 19201 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:06.591410 19208 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe59704451594e3eab64d78553356c38. Latest consensus state: current_term: 1 leader_uuid: "fe59704451594e3eab64d78553356c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe59704451594e3eab64d78553356c38" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } } }
I20260620 15:21:06.591862 19208 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:06.592170 19207 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe59704451594e3eab64d78553356c38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe59704451594e3eab64d78553356c38" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } } }
I20260620 15:21:06.592646 19207 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:06.594120 19215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:06.599555 19215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:06.608234 19215 catalog_manager.cc:1347] Generated new cluster ID: af24d675946c4ee8abdefe0dd3d77632
I20260620 15:21:06.608350 19215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:06.622433 19215 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:06.623759 19215 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:06.639930 19215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38: Generated new TSK 0
I20260620 15:21:06.640741 19215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:06.649447 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--builtin_ntp_servers=127.0.97.84:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:21:06.965092 19225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:06.965335 19225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:06.965418 19225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:06.972055 19225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:06.972133 19225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:06.972250 19225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:21:06.980795 19225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:06.981880 19225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:06.983466 19225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:06.996348 19233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:06.996321 19231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:06.996451 19230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:06.997279 19225 server_base.cc:1034] running on GCE node
I20260620 15:21:06.997787 19225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:06.998588 19225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:06.999805 19225 hybrid_clock.cc:648] HybridClock initialized: now 1781968866999751 us; error 43 us; skew 500 ppm
I20260620 15:21:07.000119 19225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:07.002380 19225 webserver.cc:460] Webserver started at http://127.0.97.65:45377/ using document root <none> and password file <none>
I20260620 15:21:07.003057 19225 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:07.003145 19225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:07.003513 19225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:07.006500 19225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "7b74b11c7f4f45b3bc0d368a88641a4d"
format_stamp: "Formatted at 2026-06-20 15:21:06 on dist-test-slave-7s64"
server_key: "1af971330e04bb49f0c77160cc2ebefb"
server_key_iv: "08eb2220d39a2dd81805670871cc3075"
server_key_version: "encryptionkey@0"
I20260620 15:21:07.007263 19225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "7b74b11c7f4f45b3bc0d368a88641a4d"
format_stamp: "Formatted at 2026-06-20 15:21:06 on dist-test-slave-7s64"
server_key: "1af971330e04bb49f0c77160cc2ebefb"
server_key_iv: "08eb2220d39a2dd81805670871cc3075"
server_key_version: "encryptionkey@0"
I20260620 15:21:07.012064 19225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260620 15:21:07.015424 19239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:07.016703 19225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:21:07.016858 19225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "7b74b11c7f4f45b3bc0d368a88641a4d"
format_stamp: "Formatted at 2026-06-20 15:21:06 on dist-test-slave-7s64"
server_key: "1af971330e04bb49f0c77160cc2ebefb"
server_key_iv: "08eb2220d39a2dd81805670871cc3075"
server_key_version: "encryptionkey@0"
I20260620 15:21:07.017053 19225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:07.035825 19225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:07.036732 19225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:07.036988 19225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:07.037885 19225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:07.039357 19225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:07.039449 19225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:07.039534 19225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:07.039609 19225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:07.073887 19225 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:35115
I20260620 15:21:07.073902 19351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:35115 every 8 connection(s)
I20260620 15:21:07.075280 19225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:21:07.084842 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 19225
I20260620 15:21:07.085021 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:21:07.085420 389 external_mini_cluster.cc:1428] Setting key 30d35b19242e9163daed5b4ae60494d1
I20260620 15:21:07.088202 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:07.077826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:46335 (local address 127.0.97.126:37123)
0620 15:21:07.078236 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:07.078241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:07.079696 (+ 1455us) server_negotiation.cc:407] Connection header received
0620 15:21:07.081564 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:07.081569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:07.081644 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:07.081740 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:07.082710 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.083413 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.084929 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.085420 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.085553 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:07.087018 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:07.087024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:07.087028 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:07.087046 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:07.087140 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:07.087160 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:07.087260 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:07.087752 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 15:21:07.087842 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":209,"threads_started":1}
I20260620 15:21:07.088644 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--builtin_ntp_servers=127.0.97.84:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:21:07.089119 19355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:07.078398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37123 (local address 127.0.97.65:46335)
0620 15:21:07.079531 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:07.079583 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:07.081311 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:07.081938 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:07.081946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:07.082029 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:07.082558 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:07.082572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.083529 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.083532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:07.084776 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:07.084788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.085631 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.085637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:07.085881 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:07.086665 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:21:07.086669 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:07.086715 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:21:07.086809 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:07.087528 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:07.087536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:07.087542 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:21:07.087683 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 15:21:07.087990 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":362,"threads_started":1}
I20260620 15:21:07.090585 19352 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37123
I20260620 15:21:07.090876 19352 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:07.091539 19352 heartbeater.cc:511] Master 127.0.97.126:37123 requested a full tablet report, sending...
I20260620 15:21:07.093612 19166 ts_manager.cc:194] Registered new tserver with Master: 7b74b11c7f4f45b3bc0d368a88641a4d (127.0.97.65:35115)
I20260620 15:21:07.095304 19166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:46335
W20260620 15:21:07.412273 19356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:07.412575 19356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:07.412675 19356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:07.419448 19356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:07.419526 19356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:07.419646 19356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:21:07.428371 19356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:07.429459 19356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:07.431087 19356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:07.443511 19361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:07.443562 19364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:07.443529 19362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:07.444326 19356 server_base.cc:1034] running on GCE node
I20260620 15:21:07.444922 19356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:07.445763 19356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:07.446986 19356 hybrid_clock.cc:648] HybridClock initialized: now 1781968867446942 us; error 58 us; skew 500 ppm
I20260620 15:21:07.447278 19356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:07.449399 19356 webserver.cc:460] Webserver started at http://127.0.97.66:35921/ using document root <none> and password file <none>
I20260620 15:21:07.450043 19356 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:07.450169 19356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:07.450481 19356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:07.453125 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "e6a4eb8b8be94741b095bb89c81e7629"
format_stamp: "Formatted at 2026-06-20 15:21:07 on dist-test-slave-7s64"
server_key: "c139a1ea37339ba89c94ce0a2dd31de3"
server_key_iv: "c95027631b1eaef349bbda2e51430756"
server_key_version: "encryptionkey@0"
I20260620 15:21:07.453873 19356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "e6a4eb8b8be94741b095bb89c81e7629"
format_stamp: "Formatted at 2026-06-20 15:21:07 on dist-test-slave-7s64"
server_key: "c139a1ea37339ba89c94ce0a2dd31de3"
server_key_iv: "c95027631b1eaef349bbda2e51430756"
server_key_version: "encryptionkey@0"
I20260620 15:21:07.458474 19356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:21:07.461776 19370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:07.462924 19356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:21:07.463083 19356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "e6a4eb8b8be94741b095bb89c81e7629"
format_stamp: "Formatted at 2026-06-20 15:21:07 on dist-test-slave-7s64"
server_key: "c139a1ea37339ba89c94ce0a2dd31de3"
server_key_iv: "c95027631b1eaef349bbda2e51430756"
server_key_version: "encryptionkey@0"
I20260620 15:21:07.463282 19356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:07.490687 19356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:07.491418 19356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:07.491680 19356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:07.492614 19356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:07.494053 19356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:07.494170 19356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:07.494293 19356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:07.494380 19356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:07.527148 19356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:45233
I20260620 15:21:07.527223 19482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:45233 every 8 connection(s)
I20260620 15:21:07.528311 19356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:21:07.531116 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 19356
I20260620 15:21:07.531301 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:21:07.531738 389 external_mini_cluster.cc:1428] Setting key eb138bc01d19b182b6bee42007f937c9
I20260620 15:21:07.534477 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--builtin_ntp_servers=127.0.97.84:35987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:21:07.541352 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:07.530803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52729 (local address 127.0.97.126:37123)
0620 15:21:07.531012 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:07.531018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:07.532470 (+ 1452us) server_negotiation.cc:407] Connection header received
0620 15:21:07.534244 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:07.534253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:07.534339 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:07.534449 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:07.535479 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.536691 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.538242 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.538956 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.539094 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:07.540386 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:07.540395 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:07.540403 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:07.540429 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:07.540470 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:07.540511 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:07.540606 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:07.540813 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 15:21:07.540914 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:21:07.542004 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:07.531303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37123 (local address 127.0.97.66:52729)
0620 15:21:07.532331 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:07.532372 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:07.533994 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:07.534681 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:07.534693 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:07.534788 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:07.535301 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:07.535325 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.536867 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.536872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:07.538099 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:07.538110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.539151 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.539156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:07.539384 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:07.540076 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:21:07.540079 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:07.540117 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:21:07.540182 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:07.540609 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:07.540616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:07.540622 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:21:07.540761 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:21:07.541037 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":337,"threads_started":1}
I20260620 15:21:07.543325 19483 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37123
I20260620 15:21:07.543620 19483 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:07.544308 19483 heartbeater.cc:511] Master 127.0.97.126:37123 requested a full tablet report, sending...
I20260620 15:21:07.545763 19166 ts_manager.cc:194] Registered new tserver with Master: e6a4eb8b8be94741b095bb89c81e7629 (127.0.97.66:45233)
I20260620 15:21:07.546921 19166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:52729
W20260620 15:21:07.855370 19486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:07.855602 19486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:07.855679 19486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:07.862463 19486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:07.862548 19486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:07.862643 19486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:21:07.871433 19486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:07.872560 19486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:07.874178 19486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:07.886945 19491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:07.886904 19492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:07.886993 19494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:07.887933 19486 server_base.cc:1034] running on GCE node
I20260620 15:21:07.888453 19486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:07.889390 19486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:07.890573 19486 hybrid_clock.cc:648] HybridClock initialized: now 1781968867890545 us; error 71 us; skew 500 ppm
I20260620 15:21:07.890877 19486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:07.893020 19486 webserver.cc:460] Webserver started at http://127.0.97.67:44923/ using document root <none> and password file <none>
I20260620 15:21:07.893661 19486 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:07.893743 19486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:07.894042 19486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:07.896808 19486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "81dfb6e4e9234805a263918d34f53f8e"
format_stamp: "Formatted at 2026-06-20 15:21:07 on dist-test-slave-7s64"
server_key: "4399d0ff31647416938ae6c216cccdcf"
server_key_iv: "a053efa0ac20b772901bc8eafaad7d04"
server_key_version: "encryptionkey@0"
I20260620 15:21:07.897521 19486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "81dfb6e4e9234805a263918d34f53f8e"
format_stamp: "Formatted at 2026-06-20 15:21:07 on dist-test-slave-7s64"
server_key: "4399d0ff31647416938ae6c216cccdcf"
server_key_iv: "a053efa0ac20b772901bc8eafaad7d04"
server_key_version: "encryptionkey@0"
I20260620 15:21:07.902287 19486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:21:07.905818 19500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:07.907341 19486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:21:07.907545 19486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "81dfb6e4e9234805a263918d34f53f8e"
format_stamp: "Formatted at 2026-06-20 15:21:07 on dist-test-slave-7s64"
server_key: "4399d0ff31647416938ae6c216cccdcf"
server_key_iv: "a053efa0ac20b772901bc8eafaad7d04"
server_key_version: "encryptionkey@0"
I20260620 15:21:07.907830 19486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:07.923606 19486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:07.924415 19486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:07.924793 19486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:07.925833 19486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:07.927520 19486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:07.927672 19486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:07.927832 19486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:07.927937 19486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:07.964426 19486 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:42955
I20260620 15:21:07.964466 19612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:42955 every 8 connection(s)
I20260620 15:21:07.965860 19486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:21:07.970242 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 19486
I20260620 15:21:07.970402 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:21:07.970826 389 external_mini_cluster.cc:1428] Setting key 69b3fad51b4e5e3cb9a0cce83ce6e7e5
I20260620 15:21:07.979657 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:07.968478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:60939 (local address 127.0.97.126:37123)
0620 15:21:07.968728 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:07.968734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:07.970074 (+ 1340us) server_negotiation.cc:407] Connection header received
0620 15:21:07.972526 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:07.972534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:07.972643 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:07.972778 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:07.973929 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.974792 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.976548 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.977031 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.977213 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:07.978662 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:07.978673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:07.978685 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:07.978705 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:07.978739 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:07.978754 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:07.978863 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:07.979134 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 15:21:07.979243 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 15:21:07.980679 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:07.968967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37123 (local address 127.0.97.67:60939)
0620 15:21:07.969917 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:07.969960 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:07.972199 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:07.973026 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:07.973034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:07.973126 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:07.973738 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:07.973756 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.974955 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.974964 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:07.976354 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:07.976368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.977293 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.977301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:07.977549 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:07.978323 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:21:07.978326 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:07.978364 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:21:07.978436 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:07.978877 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:07.978891 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:07.978908 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 15:21:07.979107 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 15:21:07.979489 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":296,"threads_started":1}
I20260620 15:21:07.981935 19613 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37123
I20260620 15:21:07.982204 19613 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:07.982887 19613 heartbeater.cc:511] Master 127.0.97.126:37123 requested a full tablet report, sending...
I20260620 15:21:07.984206 19166 ts_manager.cc:194] Registered new tserver with Master: 81dfb6e4e9234805a263918d34f53f8e (127.0.97.67:42955)
I20260620 15:21:07.985095 19166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:60939
I20260620 15:21:07.985886 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:21:07.995504 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:07.989794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.0.97.126:37123)
0620 15:21:07.989987 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:07.989993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:07.990500 (+ 507us) server_negotiation.cc:407] Connection header received
0620 15:21:07.990615 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:07.990622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:07.990690 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:07.990770 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:07.991182 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.991836 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.993231 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:07.993714 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:07.993824 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:07.994840 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:07.994844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:07.994848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:07.994863 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:07.994890 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:07.994901 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:07.995005 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:07.995243 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 15:21:07.995297 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 15:21:07.999264 19165 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:21:08.007332 19165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:21:08.039721 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.018687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:45233 (local address 127.0.0.1:33362)
0620 15:21:08.030167 (+ 11480us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.030217 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:08.030393 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:08.032099 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:08.032108 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:08.032166 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:08.032607 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:08.032620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.034581 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.034585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:08.037008 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:08.037019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.038705 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.038712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:08.038949 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.039005 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 15:21:08.039122 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 15:21:08.039303 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11201,"mutex_wait_us":76,"spinlock_wait_cycles":13568,"thread_start_us":13068,"threads_started":1}
I20260620 15:21:08.039726 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.018951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.0.97.66:45233)
0620 15:21:08.030975 (+ 12024us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:08.030986 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:08.031019 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:21:08.031130 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:08.031141 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:08.031399 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:08.031594 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:08.032833 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.034389 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.037294 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.038498 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.038638 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.038908 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:08.039202 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 15:21:08.039383 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11829,"thread_start_us":12888,"threads_started":1}
I20260620 15:21:08.040464 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.018926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:35115 (local address 127.0.0.1:51256)
0620 15:21:08.032091 (+ 13165us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.032107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:08.032205 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:08.033280 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:08.033287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:08.033317 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:08.033657 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:08.033668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.035354 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.035362 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:08.037241 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:08.037255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.039715 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.039723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:08.039891 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.039920 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:21:08.040003 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 15:21:08.040113 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12987,"mutex_wait_us":24,"thread_start_us":10779,"threads_started":1}
I20260620 15:21:08.040630 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.019939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.0.97.65:35115)
0620 15:21:08.032725 (+ 12786us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:08.032736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:08.032763 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:21:08.032853 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:08.032859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:08.033091 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:08.033286 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:08.033926 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.035228 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.037472 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.039242 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.039356 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.039572 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:08.040046 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 15:21:08.040205 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12610,"thread_start_us":12522,"threads_started":1}
I20260620 15:21:08.043447 19418 tablet_service.cc:1467] Processing CreateTablet for tablet 05377485491d4603ba52e0170fcfc320 (DEFAULT_TABLE table=test-table [id=68e58060daed4810b2e1782f9c31459d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:08.043715 19287 tablet_service.cc:1467] Processing CreateTablet for tablet 05377485491d4603ba52e0170fcfc320 (DEFAULT_TABLE table=test-table [id=68e58060daed4810b2e1782f9c31459d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:08.044703 19418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05377485491d4603ba52e0170fcfc320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:08.044934 19287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05377485491d4603ba52e0170fcfc320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:08.045368 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.019210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:42955 (local address 127.0.0.1:57918)
0620 15:21:08.033036 (+ 13826us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.033051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:08.033188 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:08.034064 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:08.034069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:08.034091 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:08.034530 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:08.034541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.036472 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.036478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:08.038855 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:08.038869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.044649 (+ 5780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.044659 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:08.044832 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.044861 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:21:08.044939 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 15:21:08.045048 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13721,"mutex_wait_us":4,"thread_start_us":13603,"threads_started":1}
I20260620 15:21:08.046509 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.020482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.0.97.67:42955)
0620 15:21:08.032604 (+ 12122us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:08.032612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:08.033153 (+ 541us) server_negotiation.cc:407] Connection header received
0620 15:21:08.033386 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:08.033394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:08.033616 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:08.033786 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:08.034723 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.036287 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.039025 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.041337 (+ 2312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.045653 (+ 4316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.045906 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:08.045970 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 15:21:08.046122 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11906,"thread_start_us":11774,"threads_started":1}
I20260620 15:21:08.049957 19548 tablet_service.cc:1467] Processing CreateTablet for tablet 05377485491d4603ba52e0170fcfc320 (DEFAULT_TABLE table=test-table [id=68e58060daed4810b2e1782f9c31459d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:08.051357 19548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05377485491d4603ba52e0170fcfc320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:08.055791 19631 tablet_bootstrap.cc:492] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: Bootstrap starting.
I20260620 15:21:08.059381 19631 tablet_bootstrap.cc:654] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:08.060312 19631 log.cc:824] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:08.065301 19632 tablet_bootstrap.cc:492] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d: Bootstrap starting.
I20260620 15:21:08.067098 19631 tablet_bootstrap.cc:492] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: No bootstrap required, opened a new log
I20260620 15:21:08.067428 19631 ts_tablet_manager.cc:1397] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260620 15:21:08.069690 19634 tablet_bootstrap.cc:492] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: Bootstrap starting.
I20260620 15:21:08.070567 19632 tablet_bootstrap.cc:654] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:08.071717 19632 log.cc:824] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:08.072649 19634 tablet_bootstrap.cc:654] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:08.073803 19634 log.cc:824] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:08.074901 19632 tablet_bootstrap.cc:492] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d: No bootstrap required, opened a new log
I20260620 15:21:08.075215 19632 ts_tablet_manager.cc:1397] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 15:21:08.077939 19631 raft_consensus.cc:348] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 } } peers { permanent_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35115 } } peers { permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 } }
I20260620 15:21:08.078303 19631 raft_consensus.cc:374] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:08.078377 19631 raft_consensus.cc:729] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6a4eb8b8be94741b095bb89c81e7629, State: Initialized, Role: FOLLOWER
I20260620 15:21:08.078951 19631 consensus_queue.cc:260] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 } } peers { permanent_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35115 } } peers { permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 } }
I20260620 15:21:08.080693 19631 ts_tablet_manager.cc:1428] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260620 15:21:08.081571 19483 heartbeater.cc:503] Master 127.0.97.126:37123 was elected leader, sending a full tablet report...
I20260620 15:21:08.082965 19634 tablet_bootstrap.cc:492] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: No bootstrap required, opened a new log
I20260620 15:21:08.083204 19634 ts_tablet_manager.cc:1397] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260620 15:21:08.088011 19632 raft_consensus.cc:348] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 } } peers { permanent_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35115 } } peers { permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 } }
I20260620 15:21:08.088405 19632 raft_consensus.cc:374] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:08.088542 19632 raft_consensus.cc:729] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b74b11c7f4f45b3bc0d368a88641a4d, State: Initialized, Role: FOLLOWER
I20260620 15:21:08.089156 19632 consensus_queue.cc:260] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 } } peers { permanent_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35115 } } peers { permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 } }
I20260620 15:21:08.090181 19352 heartbeater.cc:503] Master 127.0.97.126:37123 was elected leader, sending a full tablet report...
I20260620 15:21:08.090749 19632 ts_tablet_manager.cc:1428] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 15:21:08.092509 19634 raft_consensus.cc:348] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 } } peers { permanent_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35115 } } peers { permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 } }
I20260620 15:21:08.092942 19634 raft_consensus.cc:374] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:08.093042 19634 raft_consensus.cc:729] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81dfb6e4e9234805a263918d34f53f8e, State: Initialized, Role: FOLLOWER
I20260620 15:21:08.093631 19634 consensus_queue.cc:260] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 } } peers { permanent_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35115 } } peers { permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 } }
I20260620 15:21:08.094771 19613 heartbeater.cc:503] Master 127.0.97.126:37123 was elected leader, sending a full tablet report...
I20260620 15:21:08.095141 19634 ts_tablet_manager.cc:1428] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260620 15:21:08.158761 19638 raft_consensus.cc:482] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:21:08.159061 19638 raft_consensus.cc:504] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 } } peers { permanent_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35115 } } peers { permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 } }
I20260620 15:21:08.160876 19638 leader_election.cc:288] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6a4eb8b8be94741b095bb89c81e7629 (127.0.97.66:45233), 81dfb6e4e9234805a263918d34f53f8e (127.0.97.67:42955)
I20260620 15:21:08.168311 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.161520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47311 (local address 127.0.97.67:42955)
0620 15:21:08.161816 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:08.161824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:08.162822 (+ 998us) server_negotiation.cc:407] Connection header received
0620 15:21:08.163017 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:08.163021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:08.163100 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:08.163188 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:08.163918 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.164742 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.166569 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.167436 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.167600 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.167712 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:08.167959 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 15:21:08.168076 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 15:21:08.168421 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.161824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:42955 (local address 127.0.97.65:47311)
0620 15:21:08.162708 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.162725 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:08.162883 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:08.163227 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:08.163233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:08.163268 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:08.163719 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:08.163727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.164880 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.164884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:08.166396 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:08.166406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.167629 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.167635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:08.167808 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.167830 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:21:08.167934 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:21:08.168054 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":36,"thread_start_us":563,"threads_started":1}
I20260620 15:21:08.168468 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.161236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:40559 (local address 127.0.97.66:45233)
0620 15:21:08.161449 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:08.161456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:08.162686 (+ 1230us) server_negotiation.cc:407] Connection header received
0620 15:21:08.162925 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:08.162932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:08.163015 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:08.163096 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:08.163818 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.164582 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.166674 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.167552 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.167670 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.167779 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:08.168113 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 15:21:08.168231 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:21:08.168643 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.161454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:45233 (local address 127.0.97.65:40559)
0620 15:21:08.162550 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.162574 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:08.162727 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:08.163146 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:08.163152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:08.163188 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:08.163657 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:08.163668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.164718 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.164723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:08.166433 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:08.166451 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.167731 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.167737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:08.167887 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.167909 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:21:08.168083 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 15:21:08.168201 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":135,"spinlock_wait_cycles":21888,"thread_start_us":510,"threads_started":1}
I20260620 15:21:08.169562 19438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05377485491d4603ba52e0170fcfc320" candidate_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a4eb8b8be94741b095bb89c81e7629" is_pre_election: true
I20260620 15:21:08.169711 19568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05377485491d4603ba52e0170fcfc320" candidate_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dfb6e4e9234805a263918d34f53f8e" is_pre_election: true
I20260620 15:21:08.170189 19438 raft_consensus.cc:2454] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b74b11c7f4f45b3bc0d368a88641a4d in term 0.
I20260620 15:21:08.170205 19568 raft_consensus.cc:2454] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b74b11c7f4f45b3bc0d368a88641a4d in term 0.
I20260620 15:21:08.170830 19241 leader_election.cc:302] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b74b11c7f4f45b3bc0d368a88641a4d, 81dfb6e4e9234805a263918d34f53f8e; no voters:
I20260620 15:21:08.171175 19638 raft_consensus.cc:2781] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:21:08.171322 19638 raft_consensus.cc:482] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:21:08.171428 19638 raft_consensus.cc:3037] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:08.173586 19638 raft_consensus.cc:504] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 } } peers { permanent_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35115 } } peers { permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 } }
I20260620 15:21:08.174135 19638 leader_election.cc:288] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [CANDIDATE]: Term 1 election: Requested vote from peers e6a4eb8b8be94741b095bb89c81e7629 (127.0.97.66:45233), 81dfb6e4e9234805a263918d34f53f8e (127.0.97.67:42955)
I20260620 15:21:08.174530 19438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05377485491d4603ba52e0170fcfc320" candidate_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6a4eb8b8be94741b095bb89c81e7629"
I20260620 15:21:08.174599 19568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05377485491d4603ba52e0170fcfc320" candidate_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81dfb6e4e9234805a263918d34f53f8e"
I20260620 15:21:08.174721 19438 raft_consensus.cc:3037] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:08.174846 19568 raft_consensus.cc:3037] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:08.177032 19438 raft_consensus.cc:2454] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b74b11c7f4f45b3bc0d368a88641a4d in term 1.
I20260620 15:21:08.177081 19568 raft_consensus.cc:2454] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b74b11c7f4f45b3bc0d368a88641a4d in term 1.
I20260620 15:21:08.177443 19240 leader_election.cc:302] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b74b11c7f4f45b3bc0d368a88641a4d, e6a4eb8b8be94741b095bb89c81e7629; no voters:
I20260620 15:21:08.177739 19638 raft_consensus.cc:2781] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:08.178097 19638 raft_consensus.cc:686] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 1 LEADER]: Becoming Leader. State: Replica: 7b74b11c7f4f45b3bc0d368a88641a4d, State: Running, Role: LEADER
I20260620 15:21:08.178539 19638 consensus_queue.cc:237] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 } } peers { permanent_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35115 } } peers { permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 } }
I20260620 15:21:08.182744 19165 catalog_manager.cc:5495] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b74b11c7f4f45b3bc0d368a88641a4d (127.0.97.65). New cstate: current_term: 1 leader_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b74b11c7f4f45b3bc0d368a88641a4d" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 35115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 } health_report { overall_health: UNKNOWN } } }
W20260620 15:21:08.217722 19614 tablet.cc:2369] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:21:08.232084 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.226886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.0.97.65:35115)
0620 15:21:08.227053 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:08.227059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:08.227075 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:21:08.227368 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:08.227372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:08.227462 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:08.227579 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:08.227995 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.228729 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.230167 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.230564 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.230648 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.231109 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:08.231544 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:08.231667 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:08.231845 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 15:21:08.231887 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:21:08.239773 19438 raft_consensus.cc:1261] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629 [term 1 FOLLOWER]: Refusing update from remote peer 7b74b11c7f4f45b3bc0d368a88641a4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:08.239776 19568 raft_consensus.cc:1261] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e [term 1 FOLLOWER]: Refusing update from remote peer 7b74b11c7f4f45b3bc0d368a88641a4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:08.240685 19642 consensus_queue.cc:1035] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "81dfb6e4e9234805a263918d34f53f8e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 42955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:08.241055 19638 consensus_queue.cc:1035] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6a4eb8b8be94741b095bb89c81e7629" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:08.265470 19165 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:21:08.265699 19165 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:21:08.269135 19165 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe59704451594e3eab64d78553356c38: Sending DeleteTablet for 3 replicas of tablet 05377485491d4603ba52e0170fcfc320
I20260620 15:21:08.270169 19418 tablet_service.cc:1514] Processing DeleteTablet for tablet 05377485491d4603ba52e0170fcfc320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:08 UTC) from {username='slave'} at 127.0.0.1:33362
I20260620 15:21:08.270164 19287 tablet_service.cc:1514] Processing DeleteTablet for tablet 05377485491d4603ba52e0170fcfc320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:08 UTC) from {username='slave'} at 127.0.0.1:51256
I20260620 15:21:08.270512 19548 tablet_service.cc:1514] Processing DeleteTablet for tablet 05377485491d4603ba52e0170fcfc320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:08 UTC) from {username='slave'} at 127.0.0.1:57918
I20260620 15:21:08.272156 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 19225
I20260620 15:21:08.284507 19657 tablet_replica.cc:330] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: stopping tablet replica
I20260620 15:21:08.285007 19657 raft_consensus.cc:2229] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:21:08.285518 19657 raft_consensus.cc:2258] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:08.289510 19655 tablet_replica.cc:330] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d: stopping tablet replica
I20260620 15:21:08.289924 19655 raft_consensus.cc:2229] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:21:08.290380 19657 ts_tablet_manager.cc:1905] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:08.290521 19655 raft_consensus.cc:2258] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:08.292534 19655 ts_tablet_manager.cc:1905] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:08.294303 19656 tablet_replica.cc:330] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: stopping tablet replica
I20260620 15:21:08.294670 19656 raft_consensus.cc:2229] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:21:08.296813 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.273743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.0.97.65:35115)
0620 15:21:08.273961 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:08.273969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:08.288445 (+ 14476us) server_negotiation.cc:407] Connection header received
0620 15:21:08.288678 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:08.288690 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:08.288799 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:08.288924 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:08.289758 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.290938 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.292859 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.293486 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.293578 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.295106 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:08.295162 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:08.295178 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:08.295900 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:08.295958 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:08.295981 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:08.296060 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:08.296310 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 15:21:08.296402 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:21:08.297964 19347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:08.298503 19656 raft_consensus.cc:2258] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:08.298520 19657 ts_tablet_manager.cc:1918] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:08.298678 19657 log.cc:1196] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/05377485491d4603ba52e0170fcfc320
I20260620 15:21:08.299345 19657 ts_tablet_manager.cc:1939] T 05377485491d4603ba52e0170fcfc320 P 81dfb6e4e9234805a263918d34f53f8e: Deleting consensus metadata
I20260620 15:21:08.301004 19656 ts_tablet_manager.cc:1905] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:08.301908 19151 catalog_manager.cc:4841] TS 81dfb6e4e9234805a263918d34f53f8e (127.0.97.67:42955): tablet 05377485491d4603ba52e0170fcfc320 (table test-table [id=68e58060daed4810b2e1782f9c31459d]) successfully deleted
I20260620 15:21:08.306615 19656 ts_tablet_manager.cc:1918] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:08.306725 19656 log.cc:1196] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/05377485491d4603ba52e0170fcfc320
I20260620 15:21:08.307189 19656 ts_tablet_manager.cc:1939] T 05377485491d4603ba52e0170fcfc320 P e6a4eb8b8be94741b095bb89c81e7629: Deleting consensus metadata
I20260620 15:21:08.308460 19150 catalog_manager.cc:4841] TS e6a4eb8b8be94741b095bb89c81e7629 (127.0.97.66:45233): tablet 05377485491d4603ba52e0170fcfc320 (table test-table [id=68e58060daed4810b2e1782f9c31459d]) successfully deleted
I20260620 15:21:08.396713 19655 ts_tablet_manager.cc:1918] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:08.396955 19655 log.cc:1196] T 05377485491d4603ba52e0170fcfc320 P 7b74b11c7f4f45b3bc0d368a88641a4d: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/05377485491d4603ba52e0170fcfc320
I20260620 15:21:08.397249 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 19225
I20260620 15:21:08.415890 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 19356
W20260620 15:21:08.415971 19152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:35115 (error 108)
I20260620 15:21:08.418068 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.417563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:35115 (local address 127.0.0.1:51282)
0620 15:21:08.417772 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.417869 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260620 15:21:08.418296 19152 catalog_manager.cc:4584] TS 7b74b11c7f4f45b3bc0d368a88641a4d (127.0.97.65:35115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05377485491d4603ba52e0170fcfc320: Network error: Client connection negotiation failed: client connection to 127.0.97.65:35115: connect: Connection refused (error 111)
I20260620 15:21:08.422703 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.416772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.0.97.66:45233)
0620 15:21:08.417007 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:08.417015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:08.417036 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:21:08.417242 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:08.417250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:08.417378 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:08.417477 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:08.417953 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.418850 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.420008 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.420403 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.420658 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.421425 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:08.421432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:08.421445 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:08.421984 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:08.422074 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:08.422100 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:08.422242 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:08.422302 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 15:21:08.422378 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 15:21:08.423343 19478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:08.471835 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.471330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:35115 (local address 127.0.0.1:51296)
0620 15:21:08.471520 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.471610 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 15:21:08.473834 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.473337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:35115 (local address 127.0.0.1:51298)
0620 15:21:08.473552 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.473666 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 15:21:08.511545 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 19356
I20260620 15:21:08.528206 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 19486
I20260620 15:21:08.534754 19630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.529001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.0.97.67:42955)
0620 15:21:08.529152 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:08.529157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:08.529174 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:21:08.529338 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:08.529342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:08.529432 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:08.529514 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:08.529927 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.530608 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.531888 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:08.532249 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:08.532358 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:08.533419 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:08.533427 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:08.533442 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:08.533976 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:08.534019 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:08.534037 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:08.534125 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:08.534365 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 15:21:08.534441 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 15:21:08.535030 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.534535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:35115 (local address 127.0.0.1:51314)
0620 15:21:08.534748 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.534844 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 15:21:08.535326 19608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:08.536027 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.535694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:35115 (local address 127.0.0.1:51328)
0620 15:21:08.535836 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.535908 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 15:21:08.616979 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.616417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:35115 (local address 127.0.0.1:51342)
0620 15:21:08.616725 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.616826 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260620 15:21:08.618055 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:08.617662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:35115 (local address 127.0.0.1:51346)
0620 15:21:08.617864 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:08.617948 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:35115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 15:21:08.622581 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 19486
I20260620 15:21:08.639496 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 19135
I20260620 15:21:08.640437 19196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:08.747769 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 19135
2026-06-20T15:21:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2681 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:08 dist-test-slave-7s64 krb5kdc[19666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:08 dist-test-slave-7s64 krb5kdc[19666](info): set up 2 sockets
Jun 20 15:21:08 dist-test-slave-7s64 krb5kdc[19666](info): commencing operation
krb5kdc: starting...
W20260620 15:21:09.864638 389 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:10 dist-test-slave-7s64 krb5kdc[19666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:21:10.083470 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38931
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:10.390134 19679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:10.390435 19679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:10.390599 19679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:10.397526 19679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:10.397657 19679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:10.397739 19679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:10.397819 19679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:10.397884 19679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:10.406906 19679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:10.408056 19679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:10.409322 19679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:10.420432 19684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:10.420511 19685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:10.420398 19687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:10.421891 19679 server_base.cc:1034] running on GCE node
I20260620 15:21:10.422549 19679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:10.423786 19679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:10.425002 19679 hybrid_clock.cc:648] HybridClock initialized: now 1781968870425019 us; error 93 us; skew 500 ppm
I20260620 15:21:10.425346 19679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:10 dist-test-slave-7s64 krb5kdc[19666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:10.433357 19679 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:21:10.434772 19679 webserver.cc:460] Webserver started at http://127.0.97.126:40031/ using document root <none> and password file <none>
I20260620 15:21:10.435423 19679 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:10.435571 19679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:10.435858 19679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:10.438658 19679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "406acb5929114892bb39197c1dfc0766"
format_stamp: "Formatted at 2026-06-20 15:21:10 on dist-test-slave-7s64"
server_key: "9febaacd29b28671e2f183bb3bcdd918"
server_key_iv: "5a5afe2c2610867d642b7ffd5680315c"
server_key_version: "encryptionkey@0"
I20260620 15:21:10.439381 19679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "406acb5929114892bb39197c1dfc0766"
format_stamp: "Formatted at 2026-06-20 15:21:10 on dist-test-slave-7s64"
server_key: "9febaacd29b28671e2f183bb3bcdd918"
server_key_iv: "5a5afe2c2610867d642b7ffd5680315c"
server_key_version: "encryptionkey@0"
I20260620 15:21:10.444078 19679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 15:21:10.447299 19694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:10.448657 19679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 15:21:10.448827 19679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "406acb5929114892bb39197c1dfc0766"
format_stamp: "Formatted at 2026-06-20 15:21:10 on dist-test-slave-7s64"
server_key: "9febaacd29b28671e2f183bb3bcdd918"
server_key_iv: "5a5afe2c2610867d642b7ffd5680315c"
server_key_version: "encryptionkey@0"
I20260620 15:21:10.449018 19679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:10.470386 19679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:10.474054 19679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:10.474329 19679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:10.493340 19679 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:38931
I20260620 15:21:10.493383 19745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:38931 every 8 connection(s)
I20260620 15:21:10.494616 19679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:10.498502 19746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:10.502050 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 19679
I20260620 15:21:10.502238 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:10.502763 389 external_mini_cluster.cc:1428] Setting key b5c180e70398ac5bc8dba99111e7f332
I20260620 15:21:10.511855 19746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766: Bootstrap starting.
I20260620 15:21:10.516211 19746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:10.517736 19746 log.cc:824] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:21:10 dist-test-slave-7s64 krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:10.520990 19746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766: No bootstrap required, opened a new log
I20260620 15:21:10.527269 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:10.504649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.0.97.126:38931)
0620 15:21:10.505366 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:10.505377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:10.505420 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:21:10.507031 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:10.507052 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:10.507426 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:10.507731 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:10.509950 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:10.511220 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:10.513339 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:10.514001 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:10.514147 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:10.519885 (+ 5738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:10.519892 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:10.519925 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:10.519962 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:10.523075 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:10.523948 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:10.523954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:10.523959 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:10.524106 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:10.524628 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:10.524632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:10.524634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:10.525094 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:10.525333 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:10.525846 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 15:21:10.526167 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":338,"threads_started":1}
I20260620 15:21:10.531491 19746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406acb5929114892bb39197c1dfc0766" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38931 } }
I20260620 15:21:10.531912 19746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:10.532052 19746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 406acb5929114892bb39197c1dfc0766, State: Initialized, Role: FOLLOWER
I20260620 15:21:10.532748 19746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406acb5929114892bb39197c1dfc0766" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38931 } }
I20260620 15:21:10.533046 19746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:10.533211 19746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:10.533402 19746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:10.535737 19746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406acb5929114892bb39197c1dfc0766" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38931 } }
I20260620 15:21:10.536388 19746 leader_election.cc:302] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 406acb5929114892bb39197c1dfc0766; no voters:
I20260620 15:21:10.537089 19746 leader_election.cc:288] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:10.537199 19751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:10.537961 19751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [term 1 LEADER]: Becoming Leader. State: Replica: 406acb5929114892bb39197c1dfc0766, State: Running, Role: LEADER
I20260620 15:21:10.538494 19751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406acb5929114892bb39197c1dfc0766" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38931 } }
I20260620 15:21:10.539321 19746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:10.542639 19752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "406acb5929114892bb39197c1dfc0766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406acb5929114892bb39197c1dfc0766" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38931 } } }
I20260620 15:21:10.543378 19752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:10.542888 19753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 406acb5929114892bb39197c1dfc0766. Latest consensus state: current_term: 1 leader_uuid: "406acb5929114892bb39197c1dfc0766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "406acb5929114892bb39197c1dfc0766" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38931 } } }
I20260620 15:21:10.544183 19753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766 [sys.catalog]: This master's current role is: LEADER
W20260620 15:21:10.550208 19767 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:21:10.550333 19767 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:21:10.552162 19761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:10.554435 19761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:10.562932 19761 catalog_manager.cc:1347] Generated new cluster ID: 282100c1fa4d4d4d82a9f630bd9864ef
I20260620 15:21:10.563083 19761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:10.580664 19761 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:10.582093 19761 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:10.593284 19761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766: Generated new TSK 0
I20260620 15:21:10.594041 19761 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260620 15:21:10.659574 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38931
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:10.987314 19772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:10.987603 19772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:10.987775 19772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:10.994652 19772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:10.994786 19772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:10.994951 19772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:21:11.004616 19772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:11.005805 19772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:11.007115 19772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:11.019794 19777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:11.019863 19780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:11.019837 19778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:11.020802 19772 server_base.cc:1034] running on GCE node
I20260620 15:21:11.021385 19772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:11.022192 19772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:11.023406 19772 hybrid_clock.cc:648] HybridClock initialized: now 1781968871023361 us; error 40 us; skew 500 ppm
I20260620 15:21:11.023663 19772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:11 dist-test-slave-7s64 krb5kdc[19666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:11.026641 19772 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260620 15:21:11.028241 19772 webserver.cc:460] Webserver started at http://127.0.97.65:44593/ using document root <none> and password file <none>
I20260620 15:21:11.028935 19772 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:11.029033 19772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:11.029373 19772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:11.032193 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "ba9a61f02d784f8c93f348ef309fac65"
format_stamp: "Formatted at 2026-06-20 15:21:11 on dist-test-slave-7s64"
server_key: "89a0e4995b377484952d075e319b9350"
server_key_iv: "9a712f25b4dd2d354428b32c3f7cdb8b"
server_key_version: "encryptionkey@0"
I20260620 15:21:11.033005 19772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "ba9a61f02d784f8c93f348ef309fac65"
format_stamp: "Formatted at 2026-06-20 15:21:11 on dist-test-slave-7s64"
server_key: "89a0e4995b377484952d075e319b9350"
server_key_iv: "9a712f25b4dd2d354428b32c3f7cdb8b"
server_key_version: "encryptionkey@0"
I20260620 15:21:11.037889 19772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:21:11.041275 19787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:11.042516 19772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:21:11.042673 19772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "ba9a61f02d784f8c93f348ef309fac65"
format_stamp: "Formatted at 2026-06-20 15:21:11 on dist-test-slave-7s64"
server_key: "89a0e4995b377484952d075e319b9350"
server_key_iv: "9a712f25b4dd2d354428b32c3f7cdb8b"
server_key_version: "encryptionkey@0"
I20260620 15:21:11.042873 19772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:11.068034 19772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:11.071712 19772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:11.071985 19772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:11.072909 19772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:11.074141 19772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:11.074244 19772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:11.074422 19772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:11.074497 19772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:11.107242 19772 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:39537
I20260620 15:21:11.107327 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:39537 every 8 connection(s)
I20260620 15:21:11.108440 19772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:21:11.112800 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 19772
I20260620 15:21:11.112965 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:21:11.113516 389 external_mini_cluster.cc:1428] Setting key a38aceb3711d5eaebf072d741bb1b97a
Jun 20 15:21:11 dist-test-slave-7s64 krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:11.128770 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:11.110939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:52727 (local address 127.0.97.126:38931)
0620 15:21:11.111419 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:11.111424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:11.112163 (+ 739us) server_negotiation.cc:407] Connection header received
0620 15:21:11.114723 (+ 2560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:11.114729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:11.114850 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:11.114969 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:11.116524 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:11.117258 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:11.118772 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:11.119212 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:11.119349 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:11.123021 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:11.123025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:11.123029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:11.123060 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:11.125719 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:11.126568 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:11.126572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:11.126575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:11.126681 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:11.127127 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:11.127130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:11.127132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:11.127364 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:11.127475 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:11.128322 (+ 847us) server_negotiation.cc:299] Negotiation successful
0620 15:21:11.128475 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":281,"threads_started":1}
I20260620 15:21:11.130136 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:11.111345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38931 (local address 127.0.97.65:52727)
0620 15:21:11.111988 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:11.112033 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:11.114411 (+ 2378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:11.115221 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:11.115233 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:11.115807 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:11.116329 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:11.116345 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:11.117412 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:11.117417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:11.118606 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:11.118622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:11.119441 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:11.119448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:11.119676 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:11.120337 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:11.120368 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:11.122787 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:11.125850 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:11.125861 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:11.125879 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:11.126462 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:11.126781 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:11.126789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:11.126795 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:11.127025 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:11.127487 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:11.127496 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:11.127916 (+ 420us) client_negotiation.cc:769] Sending connection context
0620 15:21:11.128198 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 15:21:11.128771 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":271,"threads_started":1}
I20260620 15:21:11.131630 19900 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38931
I20260620 15:21:11.131948 19900 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:11.132794 19900 heartbeater.cc:511] Master 127.0.97.126:38931 requested a full tablet report, sending...
I20260620 15:21:11.136122 19711 ts_manager.cc:194] Registered new tserver with Master: ba9a61f02d784f8c93f348ef309fac65 (127.0.97.65:39537)
I20260620 15:21:11.138315 19711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:52727
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260620 15:21:11.167505 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38931
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:11.497519 19906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:11.497752 19906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:11.497900 19906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:11.504899 19906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:11.504984 19906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:11.505175 19906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:21:11.514943 19906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:11.516122 19906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:11.517431 19906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:11.532359 19911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:11.532436 19912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:11.533334 19914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:11.534355 19906 server_base.cc:1034] running on GCE node
I20260620 15:21:11.534955 19906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:11.535957 19906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:11.537156 19906 hybrid_clock.cc:648] HybridClock initialized: now 1781968871537111 us; error 42 us; skew 500 ppm
I20260620 15:21:11.537447 19906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:11 dist-test-slave-7s64 krb5kdc[19666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:11.540920 19906 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260620 15:21:11.542665 19906 webserver.cc:460] Webserver started at http://127.0.97.66:33213/ using document root <none> and password file <none>
I20260620 15:21:11.543710 19906 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:11.543854 19906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:11.544253 19906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:11.548592 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "18e28567903244f98633740f6939d057"
format_stamp: "Formatted at 2026-06-20 15:21:11 on dist-test-slave-7s64"
server_key: "5fc5758c7617b28b69e7ced89206480a"
server_key_iv: "f613bd470373d6eda85993e13b87f2f4"
server_key_version: "encryptionkey@0"
I20260620 15:21:11.549716 19906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "18e28567903244f98633740f6939d057"
format_stamp: "Formatted at 2026-06-20 15:21:11 on dist-test-slave-7s64"
server_key: "5fc5758c7617b28b69e7ced89206480a"
server_key_iv: "f613bd470373d6eda85993e13b87f2f4"
server_key_version: "encryptionkey@0"
I20260620 15:21:11.556849 19906 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 15:21:11.561677 19921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:11.563278 19906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:21:11.563434 19906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "18e28567903244f98633740f6939d057"
format_stamp: "Formatted at 2026-06-20 15:21:11 on dist-test-slave-7s64"
server_key: "5fc5758c7617b28b69e7ced89206480a"
server_key_iv: "f613bd470373d6eda85993e13b87f2f4"
server_key_version: "encryptionkey@0"
I20260620 15:21:11.563655 19906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:11.580847 19906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:11.586473 19906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:11.586786 19906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:11.587954 19906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:11.589828 19906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:11.589928 19906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:11.590056 19906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:11.590127 19906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:11.624414 19906 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:36179
I20260620 15:21:11.624414 20033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:36179 every 8 connection(s)
I20260620 15:21:11.626353 19906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:21:11.628145 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 19906
I20260620 15:21:11.628331 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:21:11.628798 389 external_mini_cluster.cc:1428] Setting key 75ef5fa65c3d98a143cde4f2b82c6220
Jun 20 15:21:11 dist-test-slave-7s64 krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
I20260620 15:21:11.663615 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:11.629499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:59535 (local address 127.0.97.126:38931)
0620 15:21:11.635884 (+ 6385us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:11.635893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:11.642636 (+ 6743us) server_negotiation.cc:407] Connection header received
0620 15:21:11.644926 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:11.644938 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:11.645048 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:11.645166 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:11.647320 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:11.648474 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:11.651052 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:11.651749 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:11.651924 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:11.656339 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:11.656346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:11.656353 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:11.656395 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:11.659925 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:11.660823 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:11.660827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:11.660830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:11.661104 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:11.661564 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:11.661568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:11.661570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:11.661904 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:11.662013 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:11.662852 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 15:21:11.663105 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6247,"thread_start_us":6146,"threads_started":1}
I20260620 15:21:11.664412 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:11.630424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38931 (local address 127.0.97.66:59535)
0620 15:21:11.642464 (+ 12040us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:11.642520 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:11.644676 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:11.645439 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:11.645452 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:11.646270 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:11.647065 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:11.647091 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:11.648671 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:11.648679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:11.650884 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:11.650901 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:11.652019 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:11.652028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:11.652346 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:11.653148 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:11.653182 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:11.656068 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:11.660073 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:11.660081 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:11.660095 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:11.660700 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:11.661229 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:11.661234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:11.661238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:11.661454 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:11.662000 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:11.662007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:11.662437 (+ 430us) client_negotiation.cc:769] Sending connection context
0620 15:21:11.662700 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 15:21:11.663173 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11687,"thread_start_us":11882,"threads_started":1}
I20260620 15:21:11.665843 20034 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38931
I20260620 15:21:11.666157 20034 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:11.666965 20034 heartbeater.cc:511] Master 127.0.97.126:38931 requested a full tablet report, sending...
I20260620 15:21:11.668479 19711 ts_manager.cc:194] Registered new tserver with Master: 18e28567903244f98633740f6939d057 (127.0.97.66:36179)
I20260620 15:21:11.669411 19711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:59535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260620 15:21:11.689370 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38931
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:12.000445 20040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:12.000802 20040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:12.000962 20040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:12.007951 20040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:12.008090 20040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:12.008278 20040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:21:12.017581 20040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:12.018813 20040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:12.020051 20040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:12.032668 20045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:12.032604 20048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:12.032650 20046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:12.033546 20040 server_base.cc:1034] running on GCE node
I20260620 15:21:12.034183 20040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:12.035097 20040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:12.036454 20040 hybrid_clock.cc:648] HybridClock initialized: now 1781968872036416 us; error 56 us; skew 500 ppm
I20260620 15:21:12.036795 20040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:12 dist-test-slave-7s64 krb5kdc[19666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:12.039884 20040 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260620 15:21:12.041383 20040 webserver.cc:460] Webserver started at http://127.0.97.67:33639/ using document root <none> and password file <none>
I20260620 15:21:12.041981 20040 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:12.042074 20040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:12.042340 20040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:12.045276 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "d75e908f666640d283bb1ddbd73aab0c"
format_stamp: "Formatted at 2026-06-20 15:21:12 on dist-test-slave-7s64"
server_key: "e0db4d118b3ec5f561e16e6932986617"
server_key_iv: "3d8ec3f33447a8031923e04b31fe004a"
server_key_version: "encryptionkey@0"
I20260620 15:21:12.045986 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "d75e908f666640d283bb1ddbd73aab0c"
format_stamp: "Formatted at 2026-06-20 15:21:12 on dist-test-slave-7s64"
server_key: "e0db4d118b3ec5f561e16e6932986617"
server_key_iv: "3d8ec3f33447a8031923e04b31fe004a"
server_key_version: "encryptionkey@0"
I20260620 15:21:12.050735 20040 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:21:12.054133 20055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:12.055310 20040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:21:12.055487 20040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "d75e908f666640d283bb1ddbd73aab0c"
format_stamp: "Formatted at 2026-06-20 15:21:12 on dist-test-slave-7s64"
server_key: "e0db4d118b3ec5f561e16e6932986617"
server_key_iv: "3d8ec3f33447a8031923e04b31fe004a"
server_key_version: "encryptionkey@0"
I20260620 15:21:12.055696 20040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:12.072037 20040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:12.075652 20040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:12.075911 20040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:12.076802 20040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:12.078034 20040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:12.078104 20040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:12.078281 20040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:12.078344 20040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:12.112099 20040 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:46133
I20260620 15:21:12.112133 20167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:46133 every 8 connection(s)
I20260620 15:21:12.113313 20040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:21:12.119611 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 20040
I20260620 15:21:12.119797 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:21:12.120247 389 external_mini_cluster.cc:1428] Setting key caf1673ba114efdf4bcb444318b24c3d
Jun 20 15:21:12 dist-test-slave-7s64 krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:12.131915 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.115638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:43539 (local address 127.0.97.126:38931)
0620 15:21:12.115821 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.115826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.116919 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 15:21:12.118706 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.118713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.118816 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.118930 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:12.120471 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.121301 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.122785 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.123205 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.123329 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.126532 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:12.126538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:12.126542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:12.126585 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:12.129018 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:12.129688 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:12.129697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:12.129701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:12.129829 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:12.130306 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:12.130311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:12.130314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:12.130584 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:12.130743 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.131372 (+ 629us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.131586 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:21:12.132776 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.116040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38931 (local address 127.0.97.67:43539)
0620 15:21:12.116755 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:12.116795 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:12.118421 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:12.119169 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:12.119176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:12.119630 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:12.120259 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.120276 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.121453 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.121459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.122636 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.122647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.123428 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.123435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.123651 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.124230 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:12.124254 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:12.126332 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:12.129161 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:12.129169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:12.129182 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:12.129564 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:12.129951 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:12.129956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:12.129960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:12.130167 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:12.130724 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:12.130731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:12.131023 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 15:21:12.131281 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 15:21:12.131738 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":253,"threads_started":1}
I20260620 15:21:12.134155 20168 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38931
I20260620 15:21:12.134438 20168 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:12.135174 20168 heartbeater.cc:511] Master 127.0.97.126:38931 requested a full tablet report, sending...
I20260620 15:21:12.136601 19710 ts_manager.cc:194] Registered new tserver with Master: d75e908f666640d283bb1ddbd73aab0c (127.0.97.67:46133)
I20260620 15:21:12.137462 19710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:43539
I20260620 15:21:12.142445 19900 heartbeater.cc:503] Master 127.0.97.126:38931 was elected leader, sending a full tablet report...
I20260620 15:21:12.147007 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:21:12.163048 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.151040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.0.97.126:38931)
0620 15:21:12.151257 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.151265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.151572 (+ 307us) server_negotiation.cc:407] Connection header received
0620 15:21:12.151784 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.151792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.151869 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.151968 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:12.153424 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.154143 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.155377 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.155797 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.155922 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.158186 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:12.158190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:12.158193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:12.158221 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:12.160570 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:12.161188 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:12.161195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:12.161199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:12.161320 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:12.161664 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:12.161670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:12.161673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:12.161947 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:12.162227 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.162518 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.162711 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:21:12.166635 19710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:21:12.173990 19710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:21:12.209133 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.187701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.0.97.67:46133)
0620 15:21:12.200810 (+ 13109us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.200823 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.201562 (+ 739us) server_negotiation.cc:407] Connection header received
0620 15:21:12.201690 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.201702 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.201970 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.202200 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:12.202924 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.204618 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.207080 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.207944 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.208090 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.208322 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.208636 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.208800 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12873,"thread_start_us":12655,"threads_started":1}
I20260620 15:21:12.209180 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.186897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:46133 (local address 127.0.0.1:48060)
0620 15:21:12.201293 (+ 14396us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:12.201311 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:12.201431 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:12.202200 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:12.202208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:12.202238 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:12.202665 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.202683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.204783 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.204788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.206929 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.206941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.208150 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.208157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.208359 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.208412 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 15:21:12.208592 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 15:21:12.208768 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14259,"mutex_wait_us":48,"thread_start_us":14118,"threads_started":1}
I20260620 15:21:12.211476 20182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.184830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:36179 (local address 127.0.0.1:49270)
0620 15:21:12.197548 (+ 12718us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:12.197612 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:12.197824 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:12.203670 (+ 5846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:12.203675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:12.203714 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:12.204159 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.204170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.206271 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.206278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.208710 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.208725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.210622 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.210631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.210809 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.210839 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:21:12.210937 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:21:12.211078 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12388,"mutex_wait_us":61,"thread_start_us":15120,"threads_started":1}
I20260620 15:21:12.211854 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.185180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39537 (local address 127.0.0.1:34206)
0620 15:21:12.199730 (+ 14550us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:12.199754 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:12.199891 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:12.203420 (+ 3529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:12.203426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:12.203453 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:12.203901 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.203912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.205750 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.205758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.208168 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.208185 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.211055 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.211061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.211242 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.211279 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:21:12.211374 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 15:21:12.211512 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14392,"mutex_wait_us":44,"thread_start_us":11872,"threads_started":1}
I20260620 15:21:12.211514 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.185015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.0.97.66:36179)
0620 15:21:12.202883 (+ 17868us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.202893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.202916 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:21:12.203006 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.203014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.203254 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.204340 (+ 1086us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:12.204588 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.206115 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.208911 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.210411 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.210563 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.210783 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.210989 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.211166 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17724,"thread_start_us":17630,"threads_started":1}
I20260620 15:21:12.212188 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.186080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.0.97.65:39537)
0620 15:21:12.202957 (+ 16877us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.202963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.202986 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:21:12.203062 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.203068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.203236 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.203435 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:12.204073 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.205619 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.208357 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.210581 (+ 2224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.210701 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.210914 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.211705 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.211872 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16677,"thread_start_us":17297,"threads_started":1}
I20260620 15:21:12.213527 20103 tablet_service.cc:1467] Processing CreateTablet for tablet b4b02e6b9ceb4701b2b763c1d8faa613 (DEFAULT_TABLE table=test-table [id=188d3985b2f74e63aa0ccf9f5d90d3af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:12.215489 20103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b02e6b9ceb4701b2b763c1d8faa613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:12.215248 19835 tablet_service.cc:1467] Processing CreateTablet for tablet b4b02e6b9ceb4701b2b763c1d8faa613 (DEFAULT_TABLE table=test-table [id=188d3985b2f74e63aa0ccf9f5d90d3af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:12.215623 19969 tablet_service.cc:1467] Processing CreateTablet for tablet b4b02e6b9ceb4701b2b763c1d8faa613 (DEFAULT_TABLE table=test-table [id=188d3985b2f74e63aa0ccf9f5d90d3af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:12.216523 19835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b02e6b9ceb4701b2b763c1d8faa613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:12.216956 19969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b02e6b9ceb4701b2b763c1d8faa613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:12.229063 20186 tablet_bootstrap.cc:492] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: Bootstrap starting.
I20260620 15:21:12.231164 20188 tablet_bootstrap.cc:492] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: Bootstrap starting.
I20260620 15:21:12.232684 20186 tablet_bootstrap.cc:654] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:12.233158 20187 tablet_bootstrap.cc:492] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: Bootstrap starting.
I20260620 15:21:12.234081 20186 log.cc:824] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:12.234594 20188 tablet_bootstrap.cc:654] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:12.235646 20188 log.cc:824] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:12.236459 20187 tablet_bootstrap.cc:654] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:12.237625 20187 log.cc:824] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:12.238184 20186 tablet_bootstrap.cc:492] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: No bootstrap required, opened a new log
I20260620 15:21:12.238497 20186 ts_tablet_manager.cc:1397] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260620 15:21:12.240654 20187 tablet_bootstrap.cc:492] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: No bootstrap required, opened a new log
I20260620 15:21:12.240909 20187 ts_tablet_manager.cc:1397] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260620 15:21:12.248549 20188 tablet_bootstrap.cc:492] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: No bootstrap required, opened a new log
I20260620 15:21:12.248826 20188 ts_tablet_manager.cc:1397] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.008s
I20260620 15:21:12.249490 20187 raft_consensus.cc:348] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e28567903244f98633740f6939d057" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36179 } } peers { permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 } } peers { permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 } }
I20260620 15:21:12.249948 20187 raft_consensus.cc:374] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:12.250114 20187 raft_consensus.cc:729] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e28567903244f98633740f6939d057, State: Initialized, Role: FOLLOWER
I20260620 15:21:12.250229 20186 raft_consensus.cc:348] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e28567903244f98633740f6939d057" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36179 } } peers { permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 } } peers { permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 } }
I20260620 15:21:12.250584 20186 raft_consensus.cc:374] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:12.250679 20186 raft_consensus.cc:729] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9a61f02d784f8c93f348ef309fac65, State: Initialized, Role: FOLLOWER
I20260620 15:21:12.250854 20187 consensus_queue.cc:260] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e28567903244f98633740f6939d057" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36179 } } peers { permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 } } peers { permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 } }
I20260620 15:21:12.251272 20186 consensus_queue.cc:260] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e28567903244f98633740f6939d057" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36179 } } peers { permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 } } peers { permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 } }
I20260620 15:21:12.252089 20034 heartbeater.cc:503] Master 127.0.97.126:38931 was elected leader, sending a full tablet report...
I20260620 15:21:12.252691 20187 ts_tablet_manager.cc:1428] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260620 15:21:12.257899 20186 ts_tablet_manager.cc:1428] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: Time spent starting tablet: real 0.019s user 0.008s sys 0.012s
I20260620 15:21:12.259171 20188 raft_consensus.cc:348] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e28567903244f98633740f6939d057" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36179 } } peers { permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 } } peers { permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 } }
I20260620 15:21:12.259584 20188 raft_consensus.cc:374] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:12.259713 20188 raft_consensus.cc:729] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75e908f666640d283bb1ddbd73aab0c, State: Initialized, Role: FOLLOWER
I20260620 15:21:12.260274 20188 consensus_queue.cc:260] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e28567903244f98633740f6939d057" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36179 } } peers { permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 } } peers { permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 } }
I20260620 15:21:12.261343 20168 heartbeater.cc:503] Master 127.0.97.126:38931 was elected leader, sending a full tablet report...
I20260620 15:21:12.261876 20188 ts_tablet_manager.cc:1428] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260620 15:21:12.276095 20192 raft_consensus.cc:482] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:21:12.276341 20192 raft_consensus.cc:504] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e28567903244f98633740f6939d057" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36179 } } peers { permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 } } peers { permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 } }
I20260620 15:21:12.277936 20192 leader_election.cc:288] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba9a61f02d784f8c93f348ef309fac65 (127.0.97.65:39537), d75e908f666640d283bb1ddbd73aab0c (127.0.97.67:46133)
I20260620 15:21:12.285180 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.278477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:37929 (local address 127.0.97.65:39537)
0620 15:21:12.278760 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.278768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.279327 (+ 559us) server_negotiation.cc:407] Connection header received
0620 15:21:12.279582 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.279589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.279671 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.279743 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:12.280352 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.281372 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.283307 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.284056 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.284164 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.284308 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.284804 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.284925 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:21:12.285202 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.278299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41643 (local address 127.0.97.67:46133)
0620 15:21:12.278511 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.278520 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.279453 (+ 933us) server_negotiation.cc:407] Connection header received
0620 15:21:12.279628 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.279631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.279689 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.279795 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:12.280554 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.281284 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.283267 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.284076 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.284255 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.284370 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.284780 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.284914 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:21:12.285411 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.278378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39537 (local address 127.0.97.66:37929)
0620 15:21:12.279226 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:12.279242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:12.279399 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:12.279787 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:12.279791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:12.279814 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:12.280194 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.280204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.281532 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.281536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.283123 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.283135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.284306 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.284314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.284627 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.284652 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 15:21:12.284766 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 15:21:12.284894 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":35,"thread_start_us":485,"threads_started":1}
I20260620 15:21:12.285512 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.278463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:46133 (local address 127.0.97.66:41643)
0620 15:21:12.279353 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:12.279368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:12.279504 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:12.279832 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:12.279837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:12.279866 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:12.280357 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.280371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.281436 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.281445 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.283085 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:12.283097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.284341 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.284353 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:12.284623 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.284647 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:21:12.284723 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 15:21:12.284875 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":73,"thread_start_us":548,"threads_started":1}
I20260620 15:21:12.286361 20123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b02e6b9ceb4701b2b763c1d8faa613" candidate_uuid: "18e28567903244f98633740f6939d057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d75e908f666640d283bb1ddbd73aab0c" is_pre_election: true
I20260620 15:21:12.286394 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b02e6b9ceb4701b2b763c1d8faa613" candidate_uuid: "18e28567903244f98633740f6939d057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9a61f02d784f8c93f348ef309fac65" is_pre_election: true
I20260620 15:21:12.286831 19855 raft_consensus.cc:2454] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18e28567903244f98633740f6939d057 in term 0.
I20260620 15:21:12.286831 20123 raft_consensus.cc:2454] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18e28567903244f98633740f6939d057 in term 0.
I20260620 15:21:12.287434 19925 leader_election.cc:302] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18e28567903244f98633740f6939d057, d75e908f666640d283bb1ddbd73aab0c; no voters:
I20260620 15:21:12.287832 20192 raft_consensus.cc:2781] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:21:12.287997 20192 raft_consensus.cc:482] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:21:12.288118 20192 raft_consensus.cc:3037] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:12.290812 20192 raft_consensus.cc:504] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e28567903244f98633740f6939d057" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36179 } } peers { permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 } } peers { permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 } }
I20260620 15:21:12.291452 20192 leader_election.cc:288] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [CANDIDATE]: Term 1 election: Requested vote from peers ba9a61f02d784f8c93f348ef309fac65 (127.0.97.65:39537), d75e908f666640d283bb1ddbd73aab0c (127.0.97.67:46133)
I20260620 15:21:12.291814 19855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b02e6b9ceb4701b2b763c1d8faa613" candidate_uuid: "18e28567903244f98633740f6939d057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9a61f02d784f8c93f348ef309fac65"
I20260620 15:21:12.291911 20123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b02e6b9ceb4701b2b763c1d8faa613" candidate_uuid: "18e28567903244f98633740f6939d057" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d75e908f666640d283bb1ddbd73aab0c"
I20260620 15:21:12.292027 19855 raft_consensus.cc:3037] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:12.292093 20123 raft_consensus.cc:3037] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:12.295434 19855 raft_consensus.cc:2454] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18e28567903244f98633740f6939d057 in term 1.
I20260620 15:21:12.295468 20123 raft_consensus.cc:2454] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18e28567903244f98633740f6939d057 in term 1.
I20260620 15:21:12.295830 19925 leader_election.cc:302] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18e28567903244f98633740f6939d057, d75e908f666640d283bb1ddbd73aab0c; no voters:
I20260620 15:21:12.296093 20192 raft_consensus.cc:2781] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:12.296584 20192 raft_consensus.cc:686] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 1 LEADER]: Becoming Leader. State: Replica: 18e28567903244f98633740f6939d057, State: Running, Role: LEADER
I20260620 15:21:12.297052 20192 consensus_queue.cc:237] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e28567903244f98633740f6939d057" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36179 } } peers { permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 } } peers { permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 } }
I20260620 15:21:12.301544 19710 catalog_manager.cc:5495] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18e28567903244f98633740f6939d057 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "18e28567903244f98633740f6939d057" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e28567903244f98633740f6939d057" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 } health_report { overall_health: UNKNOWN } } }
I20260620 15:21:12.331202 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.325613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.0.97.66:36179)
0620 15:21:12.325885 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.325894 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.325920 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:21:12.326077 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.326081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.326173 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.326300 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:12.326590 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.327340 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.328999 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.329434 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.329544 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.329989 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:12.330467 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:12.330634 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.330828 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.330911 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260620 15:21:12.339123 20123 raft_consensus.cc:1261] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c [term 1 FOLLOWER]: Refusing update from remote peer 18e28567903244f98633740f6939d057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:12.339123 19855 raft_consensus.cc:1261] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65 [term 1 FOLLOWER]: Refusing update from remote peer 18e28567903244f98633740f6939d057: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:12.340219 20197 consensus_queue.cc:1035] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75e908f666640d283bb1ddbd73aab0c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 46133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:12.340652 20192 consensus_queue.cc:1035] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9a61f02d784f8c93f348ef309fac65" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:12.352972 20200 mvcc.cc:205] Tried to move back new op lower bound from 7298944501091135488 to 7298944500932571136. Current Snapshot: MvccSnapshot[applied={T|T < 7298944501091135488}]
I20260620 15:21:12.355266 20201 mvcc.cc:205] Tried to move back new op lower bound from 7298944501091135488 to 7298944500932571136. Current Snapshot: MvccSnapshot[applied={T|T < 7298944501091135488}]
I20260620 15:21:12.355487 20202 mvcc.cc:205] Tried to move back new op lower bound from 7298944501091135488 to 7298944500932571136. Current Snapshot: MvccSnapshot[applied={T|T < 7298944501091135488}]
I20260620 15:21:12.365726 19711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:21:12.365967 19711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:21:12.369462 19711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 406acb5929114892bb39197c1dfc0766: Sending DeleteTablet for 3 replicas of tablet b4b02e6b9ceb4701b2b763c1d8faa613
I20260620 15:21:12.370515 19969 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b02e6b9ceb4701b2b763c1d8faa613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:12 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49270
I20260620 15:21:12.370644 19835 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b02e6b9ceb4701b2b763c1d8faa613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:12 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34206
I20260620 15:21:12.371398 20209 tablet_replica.cc:330] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: stopping tablet replica
I20260620 15:21:12.371670 20103 tablet_service.cc:1514] Processing DeleteTablet for tablet b4b02e6b9ceb4701b2b763c1d8faa613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:12 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:48060
I20260620 15:21:12.372026 20209 raft_consensus.cc:2229] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:21:12.372815 20209 raft_consensus.cc:2258] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:12.373387 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 19772
I20260620 15:21:12.375221 20209 ts_tablet_manager.cc:1905] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:12.382179 20211 tablet_replica.cc:330] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: stopping tablet replica
I20260620 15:21:12.382560 20211 raft_consensus.cc:2229] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:21:12.383015 20211 raft_consensus.cc:2258] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:12.384869 20209 ts_tablet_manager.cc:1918] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:12.384979 20211 ts_tablet_manager.cc:1905] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:12.385043 20209 log.cc:1196] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/b4b02e6b9ceb4701b2b763c1d8faa613
I20260620 15:21:12.385502 20209 ts_tablet_manager.cc:1939] T b4b02e6b9ceb4701b2b763c1d8faa613 P 18e28567903244f98633740f6939d057: Deleting consensus metadata
I20260620 15:21:12.386832 19697 catalog_manager.cc:4841] TS 18e28567903244f98633740f6939d057 (127.0.97.66:36179): tablet b4b02e6b9ceb4701b2b763c1d8faa613 (table test-table [id=188d3985b2f74e63aa0ccf9f5d90d3af]) successfully deleted
I20260620 15:21:12.388834 20210 tablet_replica.cc:330] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: stopping tablet replica
I20260620 15:21:12.389474 20210 raft_consensus.cc:2229] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:21:12.390062 20210 raft_consensus.cc:2258] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:12.391940 20211 ts_tablet_manager.cc:1918] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:12.392078 20211 log.cc:1196] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/b4b02e6b9ceb4701b2b763c1d8faa613
I20260620 15:21:12.392257 20210 ts_tablet_manager.cc:1905] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:12.392663 20211 ts_tablet_manager.cc:1939] T b4b02e6b9ceb4701b2b763c1d8faa613 P d75e908f666640d283bb1ddbd73aab0c: Deleting consensus metadata
I20260620 15:21:12.394235 19698 catalog_manager.cc:4841] TS d75e908f666640d283bb1ddbd73aab0c (127.0.97.67:46133): tablet b4b02e6b9ceb4701b2b763c1d8faa613 (table test-table [id=188d3985b2f74e63aa0ccf9f5d90d3af]) successfully deleted
I20260620 15:21:12.399268 20210 ts_tablet_manager.cc:1918] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:12.399421 20210 log.cc:1196] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781968566423581-389-0/minicluster-data/ts-0/wal/wals/b4b02e6b9ceb4701b2b763c1d8faa613
Jun 20 15:21:12 dist-test-slave-7s64 krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260620 15:21:12.399879 20210 ts_tablet_manager.cc:1939] T b4b02e6b9ceb4701b2b763c1d8faa613 P ba9a61f02d784f8c93f348ef309fac65: Deleting consensus metadata
I20260620 15:21:12.401328 19696 catalog_manager.cc:4841] TS ba9a61f02d784f8c93f348ef309fac65 (127.0.97.65:39537): tablet b4b02e6b9ceb4701b2b763c1d8faa613 (table test-table [id=188d3985b2f74e63aa0ccf9f5d90d3af]) successfully deleted
I20260620 15:21:12.406514 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.376171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.0.97.65:39537)
0620 15:21:12.376376 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.376385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.389119 (+ 12734us) server_negotiation.cc:407] Connection header received
0620 15:21:12.389543 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.389551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.389684 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.389784 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:12.391810 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.393031 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.395021 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.395671 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.395809 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.401058 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:12.401071 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:12.401088 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:12.401127 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:12.403943 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:12.404765 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:12.404771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:12.404773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:12.404899 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:12.405276 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:12.405279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:12.405281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:12.405588 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:12.405816 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.406089 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.406240 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:21:12.407117 19895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:12.499027 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 19772
I20260620 15:21:12.516985 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 19906
Jun 20 15:21:12 dist-test-slave-7s64 krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260620 15:21:12.534840 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.517761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.0.97.66:36179)
0620 15:21:12.517949 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.517954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.518095 (+ 141us) server_negotiation.cc:407] Connection header received
0620 15:21:12.518290 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.518294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.518353 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.518445 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:12.520110 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.520964 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.522699 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.523095 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.523178 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.527915 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:12.527929 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:12.527948 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:12.527995 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:12.531907 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:12.532611 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:12.532621 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:12.532625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:12.532794 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:12.533239 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:12.533245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:12.533248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:12.533683 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:12.534007 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.534245 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.534455 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:21:12.535673 20029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:12.645867 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 19906
I20260620 15:21:12.671283 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 20040
Jun 20 15:21:12 dist-test-slave-7s64 krb5kdc[19666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260620 15:21:12.685853 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:12.672084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.0.97.67:46133)
0620 15:21:12.672274 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:12.672280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:12.672300 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:21:12.672400 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:12.672406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:12.672513 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:12.672610 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:12.673779 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.674878 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.676076 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:12.676733 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:12.676838 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:12.681147 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:12.681159 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:12.681174 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:12.681214 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:12.683574 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:12.684163 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:12.684170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:12.684172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:12.684305 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:12.684663 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:12.684666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:12.684668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:12.684987 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:12.685227 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:12.685467 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 15:21:12.685623 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:21:12.686436 20163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:12.773558 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 20040
I20260620 15:21:12.789789 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 19679
I20260620 15:21:12.790858 19741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:12.899435 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 19679
2026-06-20T15:21:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4152 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T15:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:12Z Disabled control of system clock
I20260620 15:21:12.970197 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36305
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:21:13.286957 20220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:13.287225 20220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:13.287389 20220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:13.294337 20220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:13.294428 20220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:13.294528 20220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:13.294636 20220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:13.294724 20220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:13.303402 20220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:13.304659 20220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:13.306360 20220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:13.317858 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:13.317819 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:13.317894 20228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:13.318632 20220 server_base.cc:1034] running on GCE node
I20260620 15:21:13.319294 20220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:13.320562 20220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:13.321776 20220 hybrid_clock.cc:648] HybridClock initialized: now 1781968873321748 us; error 33 us; skew 500 ppm
I20260620 15:21:13.322070 20220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:13.324118 20220 webserver.cc:460] Webserver started at http://127.0.97.126:45283/ using document root <none> and password file <none>
I20260620 15:21:13.324817 20220 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:13.324906 20220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:13.325166 20220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:13.327880 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "69e6d91349094b9abde2fe29f79f2e79"
format_stamp: "Formatted at 2026-06-20 15:21:13 on dist-test-slave-7s64"
server_key: "1637d7ba05dedb09b3b96e6cdce6884c"
server_key_iv: "ad38f9f43661fc0afafd73dc6226d3a7"
server_key_version: "encryptionkey@0"
I20260620 15:21:13.328693 20220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "69e6d91349094b9abde2fe29f79f2e79"
format_stamp: "Formatted at 2026-06-20 15:21:13 on dist-test-slave-7s64"
server_key: "1637d7ba05dedb09b3b96e6cdce6884c"
server_key_iv: "ad38f9f43661fc0afafd73dc6226d3a7"
server_key_version: "encryptionkey@0"
I20260620 15:21:13.333539 20220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:21:13.336786 20234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:13.338057 20220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:21:13.338232 20220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "69e6d91349094b9abde2fe29f79f2e79"
format_stamp: "Formatted at 2026-06-20 15:21:13 on dist-test-slave-7s64"
server_key: "1637d7ba05dedb09b3b96e6cdce6884c"
server_key_iv: "ad38f9f43661fc0afafd73dc6226d3a7"
server_key_version: "encryptionkey@0"
I20260620 15:21:13.338440 20220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:13.354491 20220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:13.355224 20220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:13.355496 20220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:13.373622 20220 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36305
I20260620 15:21:13.373701 20285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36305 every 8 connection(s)
I20260620 15:21:13.374869 20220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:13.378110 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 20220
I20260620 15:21:13.378309 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:13.378588 20286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:13.378809 389 external_mini_cluster.cc:1428] Setting key 3c1dfd902ff4f12399934446f6cca266
I20260620 15:21:13.391619 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:13.380603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.0.97.126:36305)
0620 15:21:13.381574 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:13.381587 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:13.381630 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:21:13.383391 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:13.383410 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:13.383840 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:13.384224 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:13.384804 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:13.386087 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:13.387394 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:13.388036 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:13.388239 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:13.389474 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:13.389480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:13.389496 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:13.390022 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:13.390100 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:13.390118 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:13.390251 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:13.390463 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 15:21:13.390751 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":369,"threads_started":1}
I20260620 15:21:13.391649 20286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79: Bootstrap starting.
I20260620 15:21:13.395313 20286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:13.396334 20286 log.cc:824] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:13.398984 20286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79: No bootstrap required, opened a new log
I20260620 15:21:13.406672 20286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e6d91349094b9abde2fe29f79f2e79" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36305 } }
I20260620 15:21:13.407035 20286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:13.407151 20286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e6d91349094b9abde2fe29f79f2e79, State: Initialized, Role: FOLLOWER
I20260620 15:21:13.407709 20286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e6d91349094b9abde2fe29f79f2e79" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36305 } }
I20260620 15:21:13.408025 20286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:13.408151 20286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:13.408279 20286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:13.410387 20286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e6d91349094b9abde2fe29f79f2e79" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36305 } }
I20260620 15:21:13.410828 20286 leader_election.cc:302] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e6d91349094b9abde2fe29f79f2e79; no voters:
I20260620 15:21:13.411372 20286 leader_election.cc:288] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:13.411510 20291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:13.412210 20291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [term 1 LEADER]: Becoming Leader. State: Replica: 69e6d91349094b9abde2fe29f79f2e79, State: Running, Role: LEADER
I20260620 15:21:13.412772 20291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e6d91349094b9abde2fe29f79f2e79" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36305 } }
I20260620 15:21:13.413589 20286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:13.415570 20292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69e6d91349094b9abde2fe29f79f2e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e6d91349094b9abde2fe29f79f2e79" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36305 } } }
I20260620 15:21:13.415545 20293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e6d91349094b9abde2fe29f79f2e79. Latest consensus state: current_term: 1 leader_uuid: "69e6d91349094b9abde2fe29f79f2e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e6d91349094b9abde2fe29f79f2e79" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36305 } } }
I20260620 15:21:13.416010 20292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:13.416064 20293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:13.420408 20296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:13.424628 20296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:13.432292 20296 catalog_manager.cc:1347] Generated new cluster ID: d79263bbe38041b4b7bfe36ee1a2d582
I20260620 15:21:13.432386 20296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:13.448750 20296 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:13.449873 20296 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:13.461637 20296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79: Generated new TSK 0
I20260620 15:21:13.462357 20296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:13.469383 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36305
--builtin_ntp_servers=127.0.97.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:21:13.792240 20310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:13.792505 20310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:13.792659 20310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:13.799588 20310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:13.799669 20310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:13.799830 20310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260620 15:21:13.808650 20310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:13.809791 20310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:13.811479 20310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:13.828902 20316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:13.829041 20315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:13.828975 20318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:13.830154 20310 server_base.cc:1034] running on GCE node
I20260620 15:21:13.830736 20310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:13.831629 20310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:13.832867 20310 hybrid_clock.cc:648] HybridClock initialized: now 1781968873832818 us; error 51 us; skew 500 ppm
I20260620 15:21:13.833169 20310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:13.835332 20310 webserver.cc:460] Webserver started at http://127.0.97.65:38435/ using document root <none> and password file <none>
I20260620 15:21:13.835966 20310 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:13.836109 20310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:13.836445 20310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:13.839075 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/data/instance:
uuid: "4bcf13aec9dc47259fcd57a991d9869c"
format_stamp: "Formatted at 2026-06-20 15:21:13 on dist-test-slave-7s64"
server_key: "e5841d1fbe6e3bbbf5f3d15e63bee9b9"
server_key_iv: "4c36af6da194f3cbb6b9ce7798eb46cf"
server_key_version: "encryptionkey@0"
I20260620 15:21:13.839881 20310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/wal/instance:
uuid: "4bcf13aec9dc47259fcd57a991d9869c"
format_stamp: "Formatted at 2026-06-20 15:21:13 on dist-test-slave-7s64"
server_key: "e5841d1fbe6e3bbbf5f3d15e63bee9b9"
server_key_iv: "4c36af6da194f3cbb6b9ce7798eb46cf"
server_key_version: "encryptionkey@0"
I20260620 15:21:13.845140 20310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:21:13.848234 20324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:13.849538 20310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:21:13.849663 20310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/wal
uuid: "4bcf13aec9dc47259fcd57a991d9869c"
format_stamp: "Formatted at 2026-06-20 15:21:13 on dist-test-slave-7s64"
server_key: "e5841d1fbe6e3bbbf5f3d15e63bee9b9"
server_key_iv: "4c36af6da194f3cbb6b9ce7798eb46cf"
server_key_version: "encryptionkey@0"
I20260620 15:21:13.849830 20310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:13.865618 20310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:13.866253 20310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:13.866464 20310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:13.867264 20310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:13.868642 20310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:13.868726 20310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:13.868791 20310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:13.868850 20310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:13.900553 20310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:37493
I20260620 15:21:13.900609 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:37493 every 8 connection(s)
I20260620 15:21:13.901785 20310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/data/info.pb
I20260620 15:21:13.905098 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 20310
I20260620 15:21:13.905293 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-0/wal/instance
I20260620 15:21:13.905819 389 external_mini_cluster.cc:1428] Setting key cfae373594441191dfd9fb744994c393
I20260620 15:21:13.908816 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36305
--builtin_ntp_servers=127.0.97.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:21:13.915961 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:13.904086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49771 (local address 127.0.97.126:36305)
0620 15:21:13.904549 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:13.904554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:13.905971 (+ 1417us) server_negotiation.cc:407] Connection header received
0620 15:21:13.908167 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:13.908171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:13.908234 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:13.908316 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:13.909685 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:13.910401 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:13.912793 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:13.913312 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:13.913461 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:13.915213 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:13.915217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:13.915220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:13.915237 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:13.915283 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:13.915296 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:13.915367 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:13.915639 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:21:13.915736 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":213,"threads_started":1}
I20260620 15:21:13.917258 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:13.904731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36305 (local address 127.0.97.65:49771)
0620 15:21:13.905796 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:13.905862 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:13.907882 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:13.908599 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:13.908613 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:13.908733 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:13.909519 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:13.909544 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:13.910553 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:13.910560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:13.912619 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:13.912637 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:13.913588 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:13.913597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:13.913913 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:13.914808 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:21:13.914814 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:13.914863 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:21:13.914962 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:13.915395 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:13.915409 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:13.915419 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:21:13.915580 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 15:21:13.915979 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":293,"threads_started":1}
I20260620 15:21:13.918819 20437 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36305
I20260620 15:21:13.919098 20437 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:13.919907 20437 heartbeater.cc:511] Master 127.0.97.126:36305 requested a full tablet report, sending...
I20260620 15:21:13.922076 20251 ts_manager.cc:194] Registered new tserver with Master: 4bcf13aec9dc47259fcd57a991d9869c (127.0.97.65:37493)
I20260620 15:21:13.923460 20251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:49771
W20260620 15:21:14.223799 20441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:14.224045 20441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:14.224176 20441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:14.231194 20441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:14.231281 20441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:14.231441 20441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260620 15:21:14.240372 20441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:14.241655 20441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:14.243484 20441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:14.256373 20447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:14.256413 20446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:14.256331 20449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:14.257294 20441 server_base.cc:1034] running on GCE node
I20260620 15:21:14.257787 20441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:14.258569 20441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:14.259799 20441 hybrid_clock.cc:648] HybridClock initialized: now 1781968874259760 us; error 25 us; skew 500 ppm
I20260620 15:21:14.260164 20441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:14.262367 20441 webserver.cc:460] Webserver started at http://127.0.97.66:46107/ using document root <none> and password file <none>
I20260620 15:21:14.263000 20441 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:14.263078 20441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:14.263425 20441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:14.266103 20441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/data/instance:
uuid: "f4c7f3b28292403d961a2d0477821626"
format_stamp: "Formatted at 2026-06-20 15:21:14 on dist-test-slave-7s64"
server_key: "f99ffe710ee76fac1dba18aaba6324ad"
server_key_iv: "c453b1cade18874d2071a1e694732b37"
server_key_version: "encryptionkey@0"
I20260620 15:21:14.266840 20441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/wal/instance:
uuid: "f4c7f3b28292403d961a2d0477821626"
format_stamp: "Formatted at 2026-06-20 15:21:14 on dist-test-slave-7s64"
server_key: "f99ffe710ee76fac1dba18aaba6324ad"
server_key_iv: "c453b1cade18874d2071a1e694732b37"
server_key_version: "encryptionkey@0"
I20260620 15:21:14.271458 20441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:21:14.274686 20455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:14.275876 20441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 15:21:14.276034 20441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/wal
uuid: "f4c7f3b28292403d961a2d0477821626"
format_stamp: "Formatted at 2026-06-20 15:21:14 on dist-test-slave-7s64"
server_key: "f99ffe710ee76fac1dba18aaba6324ad"
server_key_iv: "c453b1cade18874d2071a1e694732b37"
server_key_version: "encryptionkey@0"
I20260620 15:21:14.276228 20441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:14.291239 20441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:14.291934 20441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:14.292196 20441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:14.293128 20441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:14.294452 20441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:14.294580 20441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:14.294708 20441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:14.294795 20441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:14.327956 20441 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:41271
I20260620 15:21:14.328040 20567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:41271 every 8 connection(s)
I20260620 15:21:14.329308 20441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/data/info.pb
I20260620 15:21:14.334550 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 20441
I20260620 15:21:14.334754 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/wal/instance
I20260620 15:21:14.335381 389 external_mini_cluster.cc:1428] Setting key d3b5d45b24cd45863790328090490e87
I20260620 15:21:14.338064 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36305
--builtin_ntp_servers=127.0.97.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:21:14.343742 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.331978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:36011 (local address 127.0.97.126:36305)
0620 15:21:14.332176 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:14.332182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:14.333676 (+ 1494us) server_negotiation.cc:407] Connection header received
0620 15:21:14.335771 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:14.335777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:14.335897 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:14.336009 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:14.337351 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.338059 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.340635 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.341351 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.341493 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.343047 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:14.343051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:14.343055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:14.343073 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:14.343105 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:14.343117 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:14.343190 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:14.343436 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 15:21:14.343513 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:21:14.345206 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.332571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36305 (local address 127.0.97.66:36011)
0620 15:21:14.333512 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:14.333547 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:14.335449 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:14.336307 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:14.336318 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:14.336437 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:14.337160 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.337181 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.338230 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.338236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.340406 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.340424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.341583 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.341592 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.341912 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.342687 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:21:14.342693 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:14.342744 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:21:14.342849 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:14.343212 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:14.343226 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:14.343237 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:21:14.343409 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 15:21:14.343819 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":314,"threads_started":1}
I20260620 15:21:14.346611 20568 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36305
I20260620 15:21:14.346935 20568 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:14.347720 20568 heartbeater.cc:511] Master 127.0.97.126:36305 requested a full tablet report, sending...
I20260620 15:21:14.349115 20251 ts_manager.cc:194] Registered new tserver with Master: f4c7f3b28292403d961a2d0477821626 (127.0.97.66:41271)
I20260620 15:21:14.349902 20251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:36011
W20260620 15:21:14.659708 20571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:14.659986 20571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:14.660136 20571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:14.667047 20571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:14.667176 20571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:14.667335 20571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260620 15:21:14.675921 20571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:14.677011 20571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:14.678678 20571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:14.691217 20576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:14.691161 20577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:14.691107 20579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:14.692394 20571 server_base.cc:1034] running on GCE node
I20260620 15:21:14.693032 20571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:14.693933 20571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:14.695163 20571 hybrid_clock.cc:648] HybridClock initialized: now 1781968874695119 us; error 55 us; skew 500 ppm
I20260620 15:21:14.695459 20571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:14.697633 20571 webserver.cc:460] Webserver started at http://127.0.97.67:34853/ using document root <none> and password file <none>
I20260620 15:21:14.698277 20571 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:14.698405 20571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:14.698707 20571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:14.701408 20571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/data/instance:
uuid: "f44f5ecb9ce943f6b4e234f449bee16d"
format_stamp: "Formatted at 2026-06-20 15:21:14 on dist-test-slave-7s64"
server_key: "83e9163cc081032fbc299169948a9075"
server_key_iv: "029f6262a6e75ea87d24cf22392a7762"
server_key_version: "encryptionkey@0"
I20260620 15:21:14.702143 20571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/wal/instance:
uuid: "f44f5ecb9ce943f6b4e234f449bee16d"
format_stamp: "Formatted at 2026-06-20 15:21:14 on dist-test-slave-7s64"
server_key: "83e9163cc081032fbc299169948a9075"
server_key_iv: "029f6262a6e75ea87d24cf22392a7762"
server_key_version: "encryptionkey@0"
I20260620 15:21:14.706715 20571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:21:14.709817 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:14.710988 20571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:21:14.711150 20571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/wal
uuid: "f44f5ecb9ce943f6b4e234f449bee16d"
format_stamp: "Formatted at 2026-06-20 15:21:14 on dist-test-slave-7s64"
server_key: "83e9163cc081032fbc299169948a9075"
server_key_iv: "029f6262a6e75ea87d24cf22392a7762"
server_key_version: "encryptionkey@0"
I20260620 15:21:14.711350 20571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:14.725486 20571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:14.726218 20571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:14.726498 20571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:14.727422 20571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:21:14.728969 20571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:21:14.729099 20571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:14.729242 20571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:21:14.729337 20571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:14.763474 20571 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:40513
I20260620 15:21:14.763504 20697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:40513 every 8 connection(s)
I20260620 15:21:14.764801 20571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/data/info.pb
I20260620 15:21:14.773838 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 20571
I20260620 15:21:14.774030 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/wal/instance
I20260620 15:21:14.774631 389 external_mini_cluster.cc:1428] Setting key a9c33c16eaab29059603bb43bea0ba5f
I20260620 15:21:14.778386 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.767240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:49463 (local address 127.0.97.126:36305)
0620 15:21:14.767447 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:14.767451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:14.768807 (+ 1356us) server_negotiation.cc:407] Connection header received
0620 15:21:14.770610 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:14.770616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:14.770711 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:14.770839 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:14.771922 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.772850 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.775225 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.775720 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.775885 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.777527 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:14.777533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:14.777538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:14.777565 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:14.777596 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:14.777607 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:14.777705 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:14.777934 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 15:21:14.778066 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:21:14.779206 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.767695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36305 (local address 127.0.97.67:49463)
0620 15:21:14.768641 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:14.768690 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:14.770344 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:14.771065 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:14.771073 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:14.771156 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:14.771711 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.771727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.773003 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.773012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.775056 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.775074 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.775953 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.775960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.776175 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.777199 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:21:14.777201 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:14.777235 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:21:14.777319 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:14.777715 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:14.777746 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:14.777759 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 15:21:14.777922 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 15:21:14.778281 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":279,"threads_started":1}
I20260620 15:21:14.780639 20698 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36305
I20260620 15:21:14.780879 20698 heartbeater.cc:464] Registering TS with master...
I20260620 15:21:14.781520 20698 heartbeater.cc:511] Master 127.0.97.126:36305 requested a full tablet report, sending...
I20260620 15:21:14.782752 20251 ts_manager.cc:194] Registered new tserver with Master: f44f5ecb9ce943f6b4e234f449bee16d (127.0.97.67:40513)
I20260620 15:21:14.783510 20251 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:49463
I20260620 15:21:14.789418 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:21:14.798735 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.793346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.0.97.126:36305)
0620 15:21:14.793559 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:14.793566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:14.793951 (+ 385us) server_negotiation.cc:407] Connection header received
0620 15:21:14.794061 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:14.794064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:14.794130 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:14.794209 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:14.794611 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.795253 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.796594 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.797037 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.797159 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.798047 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:14.798052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:14.798056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:14.798073 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:14.798108 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:14.798122 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:14.798199 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:14.798350 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 15:21:14.798452 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:21:14.802243 20251 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:21:14.809062 20251 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:21:14.830494 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.820340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37493 (local address 127.0.0.1:51300)
0620 15:21:14.821031 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:14.821073 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:14.821280 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:14.823517 (+ 2237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:14.823526 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:14.823561 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:14.824006 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.824023 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.825997 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.826005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.827787 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.827801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.829579 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.829586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.829803 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.829858 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 15:21:14.829975 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 15:21:14.830152 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":115,"thread_start_us":746,"threads_started":1}
I20260620 15:21:14.830912 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.820999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:41271 (local address 127.0.0.1:42294)
0620 15:21:14.822340 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:14.822357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:14.822475 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:14.823593 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:14.823599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:14.823627 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:14.824295 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.824324 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.826032 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.826038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.828275 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.828290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.830041 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.830048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.830233 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.830265 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:21:14.830394 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 15:21:14.830514 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1212,"mutex_wait_us":86,"thread_start_us":3371,"threads_started":1}
I20260620 15:21:14.831360 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.820926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.0.97.65:37493)
0620 15:21:14.821562 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:14.821571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:14.821599 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:21:14.821698 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:14.821709 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:14.823279 (+ 1570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:14.823498 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:14.824261 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.825847 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.827990 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.829375 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.829523 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.829736 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:14.830741 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0620 15:21:14.831015 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":1047,"threads_started":1}
I20260620 15:21:14.832306 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.821828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40513 (local address 127.0.0.1:47502)
0620 15:21:14.824646 (+ 2818us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:14.824677 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:14.824787 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:14.825563 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:14.825567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:14.825590 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:14.825890 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.825904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.827754 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.827763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.829539 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.829554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.831615 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.831622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.831782 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.831812 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:21:14.831903 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 15:21:14.832007 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2718,"thread_start_us":710,"threads_started":1}
I20260620 15:21:14.832432 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.822036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.0.97.67:40513)
0620 15:21:14.823160 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:14.823169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:14.824892 (+ 1723us) server_negotiation.cc:407] Connection header received
0620 15:21:14.824984 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:14.824991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:14.825175 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:14.825334 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:14.826061 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.827604 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.829755 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.831343 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.831478 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.831676 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:14.831938 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 15:21:14.832043 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":965,"thread_start_us":812,"threads_started":1}
I20260620 15:21:14.835464 20372 tablet_service.cc:1467] Processing CreateTablet for tablet 7d959f175e7e4a878e959cdde089522c (DEFAULT_TABLE table=test-table [id=70f22ca15ec94ee6929ac036b895c582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:14.831451 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.821511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42294 (local address 127.0.97.66:41271)
0620 15:21:14.823058 (+ 1547us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:14.823067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:14.823094 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:21:14.823205 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:14.823212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:14.823386 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:14.823835 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:14.824563 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.825873 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.828521 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.829859 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.830002 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.830247 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:14.830444 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 15:21:14.831137 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":1258,"threads_started":1}
I20260620 15:21:14.835835 20633 tablet_service.cc:1467] Processing CreateTablet for tablet 7d959f175e7e4a878e959cdde089522c (DEFAULT_TABLE table=test-table [id=70f22ca15ec94ee6929ac036b895c582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:14.836992 20372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d959f175e7e4a878e959cdde089522c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:14.837172 20633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d959f175e7e4a878e959cdde089522c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:14.839212 20503 tablet_service.cc:1467] Processing CreateTablet for tablet 7d959f175e7e4a878e959cdde089522c (DEFAULT_TABLE table=test-table [id=70f22ca15ec94ee6929ac036b895c582]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:21:14.840305 20503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d959f175e7e4a878e959cdde089522c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:14.851854 20716 tablet_bootstrap.cc:492] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c: Bootstrap starting.
I20260620 15:21:14.852881 20717 tablet_bootstrap.cc:492] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: Bootstrap starting.
I20260620 15:21:14.857882 20718 tablet_bootstrap.cc:492] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: Bootstrap starting.
I20260620 15:21:14.857915 20716 tablet_bootstrap.cc:654] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:14.859503 20717 tablet_bootstrap.cc:654] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:14.859891 20716 log.cc:824] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:14.860770 20717 log.cc:824] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:14.861766 20718 tablet_bootstrap.cc:654] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:14.862978 20718 log.cc:824] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:14.863224 20716 tablet_bootstrap.cc:492] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c: No bootstrap required, opened a new log
I20260620 15:21:14.863255 20717 tablet_bootstrap.cc:492] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: No bootstrap required, opened a new log
I20260620 15:21:14.863543 20717 ts_tablet_manager.cc:1397] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 15:21:14.863535 20716 ts_tablet_manager.cc:1397] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 15:21:14.865974 20718 tablet_bootstrap.cc:492] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: No bootstrap required, opened a new log
I20260620 15:21:14.866227 20718 ts_tablet_manager.cc:1397] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 15:21:14.870891 20717 raft_consensus.cc:348] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 } } peers { permanent_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40513 } } peers { permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 } }
I20260620 15:21:14.871220 20717 raft_consensus.cc:374] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:14.871335 20717 raft_consensus.cc:729] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44f5ecb9ce943f6b4e234f449bee16d, State: Initialized, Role: FOLLOWER
I20260620 15:21:14.871893 20717 consensus_queue.cc:260] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 } } peers { permanent_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40513 } } peers { permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 } }
I20260620 15:21:14.873046 20698 heartbeater.cc:503] Master 127.0.97.126:36305 was elected leader, sending a full tablet report...
I20260620 15:21:14.873488 20717 ts_tablet_manager.cc:1428] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260620 15:21:14.874423 20716 raft_consensus.cc:348] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 } } peers { permanent_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40513 } } peers { permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 } }
I20260620 15:21:14.874828 20716 raft_consensus.cc:374] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:14.874949 20716 raft_consensus.cc:729] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcf13aec9dc47259fcd57a991d9869c, State: Initialized, Role: FOLLOWER
I20260620 15:21:14.875514 20716 consensus_queue.cc:260] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 } } peers { permanent_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40513 } } peers { permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 } }
I20260620 15:21:14.877242 20437 heartbeater.cc:503] Master 127.0.97.126:36305 was elected leader, sending a full tablet report...
I20260620 15:21:14.877270 20716 ts_tablet_manager.cc:1428] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 15:21:14.877326 20718 raft_consensus.cc:348] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 } } peers { permanent_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40513 } } peers { permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 } }
I20260620 15:21:14.877741 20718 raft_consensus.cc:374] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:14.877846 20718 raft_consensus.cc:729] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c7f3b28292403d961a2d0477821626, State: Initialized, Role: FOLLOWER
I20260620 15:21:14.878376 20718 consensus_queue.cc:260] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 } } peers { permanent_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40513 } } peers { permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 } }
I20260620 15:21:14.879530 20568 heartbeater.cc:503] Master 127.0.97.126:36305 was elected leader, sending a full tablet report...
I20260620 15:21:14.880045 20718 ts_tablet_manager.cc:1428] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 15:21:14.899140 20722 raft_consensus.cc:482] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:21:14.899421 20722 raft_consensus.cc:504] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 } } peers { permanent_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40513 } } peers { permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 } }
I20260620 15:21:14.901161 20722 leader_election.cc:288] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bcf13aec9dc47259fcd57a991d9869c (127.0.97.65:37493), f4c7f3b28292403d961a2d0477821626 (127.0.97.66:41271)
W20260620 15:21:14.903900 20438 tablet.cc:2369] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:21:14.907958 20700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.901515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37493 (local address 127.0.97.67:49923)
0620 15:21:14.902054 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:14.902073 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:14.902235 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:14.902814 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:14.902817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:14.902836 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:14.903070 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.903082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.904679 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.904685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.906131 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.906143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.907182 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.907187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.907356 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.907410 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 15:21:14.907507 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 15:21:14.907643 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":1}
I20260620 15:21:14.907992 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.901621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:49923 (local address 127.0.97.65:37493)
0620 15:21:14.901842 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:14.901850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:14.902374 (+ 524us) server_negotiation.cc:407] Connection header received
0620 15:21:14.902447 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:14.902452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:14.902546 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:14.902632 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:14.903244 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.904541 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.906283 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.907022 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.907155 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.907269 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:14.907567 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 15:21:14.907712 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:21:14.908849 20392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d959f175e7e4a878e959cdde089522c" candidate_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcf13aec9dc47259fcd57a991d9869c" is_pre_election: true
I20260620 15:21:14.909497 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.902018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35873 (local address 127.0.97.66:41271)
0620 15:21:14.902636 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:14.902643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:14.902666 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:21:14.902740 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:14.902746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:14.902817 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:14.902915 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:21:14.903520 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.904702 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.907449 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.908684 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.908828 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.908932 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:14.909236 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 15:21:14.909335 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465}
I20260620 15:21:14.909446 20392 raft_consensus.cc:2454] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f44f5ecb9ce943f6b4e234f449bee16d in term 0.
I20260620 15:21:14.910211 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.901653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:41271 (local address 127.0.97.67:35873)
0620 15:21:14.902519 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:14.902540 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:14.902636 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:14.902949 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:14.902952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:14.902972 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:21:14.903342 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.903351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.904877 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.904884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.907252 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:14.907268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.908856 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.908862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:14.909023 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.909044 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:21:14.909719 (+ 675us) client_negotiation.cc:240] Negotiation successful
0620 15:21:14.909864 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":1,"thread_start_us":534,"threads_started":1}
I20260620 15:21:14.910194 20589 leader_election.cc:302] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcf13aec9dc47259fcd57a991d9869c, f44f5ecb9ce943f6b4e234f449bee16d; no voters:
I20260620 15:21:14.910579 20722 raft_consensus.cc:2781] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:21:14.910743 20722 raft_consensus.cc:482] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:21:14.910869 20722 raft_consensus.cc:3037] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:14.911495 20523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d959f175e7e4a878e959cdde089522c" candidate_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c7f3b28292403d961a2d0477821626" is_pre_election: true
I20260620 15:21:14.912007 20523 raft_consensus.cc:2454] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f44f5ecb9ce943f6b4e234f449bee16d in term 0.
I20260620 15:21:14.913726 20722 raft_consensus.cc:504] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 } } peers { permanent_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40513 } } peers { permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 } }
I20260620 15:21:14.914377 20722 leader_election.cc:288] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [CANDIDATE]: Term 1 election: Requested vote from peers 4bcf13aec9dc47259fcd57a991d9869c (127.0.97.65:37493), f4c7f3b28292403d961a2d0477821626 (127.0.97.66:41271)
I20260620 15:21:14.914836 20523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d959f175e7e4a878e959cdde089522c" candidate_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c7f3b28292403d961a2d0477821626"
I20260620 15:21:14.914865 20392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d959f175e7e4a878e959cdde089522c" candidate_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcf13aec9dc47259fcd57a991d9869c"
I20260620 15:21:14.915071 20392 raft_consensus.cc:3037] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:14.915035 20523 raft_consensus.cc:3037] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:14.918412 20523 raft_consensus.cc:2454] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f44f5ecb9ce943f6b4e234f449bee16d in term 1.
I20260620 15:21:14.918437 20392 raft_consensus.cc:2454] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f44f5ecb9ce943f6b4e234f449bee16d in term 1.
I20260620 15:21:14.918833 20587 leader_election.cc:302] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f44f5ecb9ce943f6b4e234f449bee16d, f4c7f3b28292403d961a2d0477821626; no voters:
I20260620 15:21:14.919123 20722 raft_consensus.cc:2781] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:14.919533 20722 raft_consensus.cc:686] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 1 LEADER]: Becoming Leader. State: Replica: f44f5ecb9ce943f6b4e234f449bee16d, State: Running, Role: LEADER
I20260620 15:21:14.919971 20722 consensus_queue.cc:237] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 } } peers { permanent_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40513 } } peers { permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 } }
I20260620 15:21:14.925065 20251 catalog_manager.cc:5495] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d reported cstate change: term changed from 0 to 1, leader changed from <none> to f44f5ecb9ce943f6b4e234f449bee16d (127.0.97.67). New cstate: current_term: 1 leader_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f44f5ecb9ce943f6b4e234f449bee16d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 } health_report { overall_health: UNKNOWN } } }
I20260620 15:21:14.954339 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.948526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.0.97.67:40513)
0620 15:21:14.948734 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:14.948740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:14.948876 (+ 136us) server_negotiation.cc:407] Connection header received
0620 15:21:14.949126 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:14.949130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:14.949216 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:14.949372 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:14.949761 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.950525 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.952022 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:14.952469 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:14.952661 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:14.953179 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:14.953753 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:14.953908 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:14.954010 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 15:21:14.954089 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:21:14.962191 20392 raft_consensus.cc:1261] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c [term 1 FOLLOWER]: Refusing update from remote peer f44f5ecb9ce943f6b4e234f449bee16d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:14.962244 20523 raft_consensus.cc:1261] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626 [term 1 FOLLOWER]: Refusing update from remote peer f44f5ecb9ce943f6b4e234f449bee16d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:14.963196 20726 consensus_queue.cc:1035] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcf13aec9dc47259fcd57a991d9869c" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:14.963527 20722 consensus_queue.cc:1035] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c7f3b28292403d961a2d0477821626" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:14.975150 20729 mvcc.cc:205] Tried to move back new op lower bound from 7298944511835439104 to 7298944511676641280. Current Snapshot: MvccSnapshot[applied={T|T < 7298944511835439104}]
I20260620 15:21:14.976653 20730 mvcc.cc:205] Tried to move back new op lower bound from 7298944511835439104 to 7298944511676641280. Current Snapshot: MvccSnapshot[applied={T|T < 7298944511835439104}]
I20260620 15:21:14.987933 20251 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:21:14.988209 20251 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:21:14.991801 20251 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69e6d91349094b9abde2fe29f79f2e79: Sending DeleteTablet for 3 replicas of tablet 7d959f175e7e4a878e959cdde089522c
I20260620 15:21:14.992867 20372 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d959f175e7e4a878e959cdde089522c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:14 UTC) from {username='slave'} at 127.0.0.1:51300
I20260620 15:21:14.992933 20503 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d959f175e7e4a878e959cdde089522c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:14 UTC) from {username='slave'} at 127.0.0.1:42294
I20260620 15:21:14.993656 20633 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d959f175e7e4a878e959cdde089522c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:21:14 UTC) from {username='slave'} at 127.0.0.1:47502
I20260620 15:21:14.993901 20738 tablet_replica.cc:330] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: stopping tablet replica
I20260620 15:21:14.994344 20738 raft_consensus.cc:2229] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:21:14.994948 20738 raft_consensus.cc:2258] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:14.995843 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 20310
I20260620 15:21:15.005298 20738 ts_tablet_manager.cc:1905] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:15.008555 20739 tablet_replica.cc:330] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c: stopping tablet replica
I20260620 15:21:15.008951 20739 raft_consensus.cc:2229] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:21:15.009462 20739 raft_consensus.cc:2258] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:15.012666 20740 tablet_replica.cc:330] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: stopping tablet replica
I20260620 15:21:15.013067 20740 raft_consensus.cc:2229] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:21:15.013746 20740 raft_consensus.cc:2258] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:21:15.016139 20740 ts_tablet_manager.cc:1905] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:15.018545 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:14.997060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51308 (local address 127.0.97.65:37493)
0620 15:21:14.997278 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:14.997286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:15.009810 (+ 12524us) server_negotiation.cc:407] Connection header received
0620 15:21:15.010141 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:15.010166 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:15.010298 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:15.010407 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:15.011290 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:15.012753 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:15.014608 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:15.015287 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:15.015387 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:15.016877 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:15.016889 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:15.016905 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:15.017622 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:15.017709 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:15.017733 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:15.017818 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:15.018071 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 15:21:15.018186 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:21:15.019670 20432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:15.021559 20738 ts_tablet_manager.cc:1918] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:15.021744 20738 log.cc:1196] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-1/wal/wals/7d959f175e7e4a878e959cdde089522c
I20260620 15:21:15.022322 20738 ts_tablet_manager.cc:1939] T 7d959f175e7e4a878e959cdde089522c P f4c7f3b28292403d961a2d0477821626: Deleting consensus metadata
I20260620 15:21:15.024245 20236 catalog_manager.cc:4841] TS f4c7f3b28292403d961a2d0477821626 (127.0.97.66:41271): tablet 7d959f175e7e4a878e959cdde089522c (table test-table [id=70f22ca15ec94ee6929ac036b895c582]) successfully deleted
I20260620 15:21:15.025242 20740 ts_tablet_manager.cc:1918] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:21:15.025550 20740 log.cc:1196] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: Deleting WAL directory at /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781968566423581-389-0/minicluster-data/ts-2/wal/wals/7d959f175e7e4a878e959cdde089522c
I20260620 15:21:15.026232 20740 ts_tablet_manager.cc:1939] T 7d959f175e7e4a878e959cdde089522c P f44f5ecb9ce943f6b4e234f449bee16d: Deleting consensus metadata
I20260620 15:21:15.028080 20238 catalog_manager.cc:4841] TS f44f5ecb9ce943f6b4e234f449bee16d (127.0.97.67:40513): tablet 7d959f175e7e4a878e959cdde089522c (table test-table [id=70f22ca15ec94ee6929ac036b895c582]) successfully deleted
I20260620 15:21:15.021016 20739 ts_tablet_manager.cc:1905] T 7d959f175e7e4a878e959cdde089522c P 4bcf13aec9dc47259fcd57a991d9869c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:21:15.114112 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 20310
W20260620 15:21:15.137631 20238 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:37493 (error 108)
I20260620 15:21:15.138525 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 20441
I20260620 15:21:15.141686 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:15.140152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37493 (local address 127.0.0.1:51318)
0620 15:21:15.141356 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:15.141518 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:37493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 15:21:15.142117 20238 catalog_manager.cc:4584] TS 4bcf13aec9dc47259fcd57a991d9869c (127.0.97.65:37493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d959f175e7e4a878e959cdde089522c: Network error: Client connection negotiation failed: client connection to 127.0.97.65:37493: connect: Connection refused (error 111)
I20260620 15:21:15.151427 20713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:15.140156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.0.97.66:41271)
0620 15:21:15.140760 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:15.140769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:15.140788 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:21:15.140870 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:15.140876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:15.140998 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:15.141123 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:15.144199 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:15.145614 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:15.147535 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:15.148156 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:15.148257 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:15.149436 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:15.149447 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:15.149465 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:15.150214 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:15.150275 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:15.150305 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:15.150409 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:15.150958 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 15:21:15.151068 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466}
I20260620 15:21:15.152120 20563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:15.195564 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:15.195116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37493 (local address 127.0.0.1:51320)
0620 15:21:15.195329 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:15.195411 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:37493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 15:21:15.196717 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:15.196308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37493 (local address 127.0.0.1:51322)
0620 15:21:15.196466 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:15.196592 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:37493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:21:15.257535 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:15.257007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37493 (local address 127.0.0.1:51338)
0620 15:21:15.257241 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:15.257347 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:37493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 15:21:15.258754 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:15.258297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37493 (local address 127.0.0.1:51346)
0620 15:21:15.258478 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:15.258573 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:37493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 15:21:15.277284 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 20441
I20260620 15:21:15.294377 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 20571
I20260620 15:21:15.301625 20715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:15.295213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.0.97.67:40513)
0620 15:21:15.295397 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:15.295405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:15.295427 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:21:15.295652 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:15.295657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:15.295762 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:15.295870 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:15.296275 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:15.297220 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:15.298520 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:15.299015 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:15.299142 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:15.300201 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:15.300214 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:15.300226 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:15.300883 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:15.300929 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:15.300947 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:15.301059 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:15.301207 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 15:21:15.301305 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:21:15.302393 20693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:15.339273 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:15.338826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37493 (local address 127.0.0.1:51358)
0620 15:21:15.339025 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:15.339123 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:37493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 15:21:15.340376 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:15.339956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37493 (local address 127.0.0.1:51374)
0620 15:21:15.340175 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:15.340248 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:37493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 15:21:15.429986 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 20571
I20260620 15:21:15.449885 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 20220
I20260620 15:21:15.450935 20281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:15.559589 20712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:15.558959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37493 (local address 127.0.0.1:51388)
0620 15:21:15.559264 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:15.559379 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:37493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 15:21:15.559552 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 20220
2026-06-20T15:21:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2656 ms)
[----------] 6 tests from EncryptionPolicyTest (20670 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:15 dist-test-slave-7s64 krb5kdc[20749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:15 dist-test-slave-7s64 krb5kdc[20749](info): set up 2 sockets
Jun 20 15:21:15 dist-test-slave-7s64 krb5kdc[20749](info): commencing operation
krb5kdc: starting...
W20260620 15:21:16.656890 389 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:16 dist-test-slave-7s64 krb5kdc[20749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:16.875521 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34541
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:17.190590 20762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:17.190903 20762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:17.191071 20762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:17.198160 20762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:17.198297 20762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:17.198392 20762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:17.198496 20762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:17.198580 20762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:17.207698 20762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34541
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:17.208874 20762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:17.210096 20762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:17.221540 20770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:17.221515 20767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:17.221679 20768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:17.222841 20762 server_base.cc:1034] running on GCE node
I20260620 15:21:17.223639 20762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:17.225028 20762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:17.226266 20762 hybrid_clock.cc:648] HybridClock initialized: now 1781968877226207 us; error 43 us; skew 500 ppm
I20260620 15:21:17.226634 20762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:17 dist-test-slave-7s64 krb5kdc[20749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:17.229789 20762 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:17.231231 20762 webserver.cc:460] Webserver started at http://127.0.0.1:44655/ using document root <none> and password file <none>
I20260620 15:21:17.231899 20762 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:17.232038 20762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:17.232363 20762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:17.235332 20762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "6c1c17b4172c4ba792d3a740bd128222"
format_stamp: "Formatted at 2026-06-20 15:21:17 on dist-test-slave-7s64"
server_key: "2603c9c25705a871a79ab61c46dfb74d"
server_key_iv: "b481c51d86c32be58e128ab1b357cdca"
server_key_version: "encryptionkey@0"
I20260620 15:21:17.236126 20762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "6c1c17b4172c4ba792d3a740bd128222"
format_stamp: "Formatted at 2026-06-20 15:21:17 on dist-test-slave-7s64"
server_key: "2603c9c25705a871a79ab61c46dfb74d"
server_key_iv: "b481c51d86c32be58e128ab1b357cdca"
server_key_version: "encryptionkey@0"
I20260620 15:21:17.241189 20762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 15:21:17.244597 20777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:17.245784 20762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260620 15:21:17.245947 20762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "6c1c17b4172c4ba792d3a740bd128222"
format_stamp: "Formatted at 2026-06-20 15:21:17 on dist-test-slave-7s64"
server_key: "2603c9c25705a871a79ab61c46dfb74d"
server_key_iv: "b481c51d86c32be58e128ab1b357cdca"
server_key_version: "encryptionkey@0"
I20260620 15:21:17.246115 20762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:17.275269 20762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:17.278941 20762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:17.279219 20762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:17.297335 20762 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34541
I20260620 15:21:17.297375 20828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34541 every 8 connection(s)
I20260620 15:21:17.298604 20762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:17.302389 20829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:17.304308 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 20762
I20260620 15:21:17.304576 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:17.305449 389 external_mini_cluster.cc:1428] Setting key 0c29e3e87d2f825b8db09c366cf59d67
I20260620 15:21:17.316560 20829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222: Bootstrap starting.
Jun 20 15:21:17 dist-test-slave-7s64 krb5kdc[20749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:21:17.319762 20829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:17.320960 20829 log.cc:824] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:17.324752 20829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222: No bootstrap required, opened a new log
I20260620 15:21:17.329982 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:17.307490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55759 (local address 127.0.0.1:34541)
0620 15:21:17.308342 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:17.308353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:17.308400 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:21:17.310162 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:17.310185 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:17.310587 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:17.310993 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:17.312547 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:17.313895 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:17.315310 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:17.316007 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:17.316254 (+ 247us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:17.320793 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:17.320802 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:17.320815 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:17.320859 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:17.325012 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:17.325843 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:17.325852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:17.325861 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:17.326076 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:17.326539 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:17.326545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:17.326549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:17.327314 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:17.327665 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:17.328087 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 15:21:17.328465 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":394,"threads_started":1}
I20260620 15:21:17.334252 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:17.335508 20829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1c17b4172c4ba792d3a740bd128222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34541 } }
I20260620 15:21:17.335883 20829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:17.336014 20829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1c17b4172c4ba792d3a740bd128222, State: Initialized, Role: FOLLOWER
I20260620 15:21:17.336719 20829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1c17b4172c4ba792d3a740bd128222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34541 } }
I20260620 15:21:17.337066 20829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:17.337248 20829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:17.337440 20829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:17.340282 20829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1c17b4172c4ba792d3a740bd128222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34541 } }
I20260620 15:21:17.340837 20829 leader_election.cc:302] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c1c17b4172c4ba792d3a740bd128222; no voters:
I20260620 15:21:17.341497 20829 leader_election.cc:288] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:17.341634 20839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:17.343020 20839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [term 1 LEADER]: Becoming Leader. State: Replica: 6c1c17b4172c4ba792d3a740bd128222, State: Running, Role: LEADER
I20260620 15:21:17.343544 20839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1c17b4172c4ba792d3a740bd128222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34541 } }
I20260620 15:21:17.343820 20829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:17.346236 20840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c1c17b4172c4ba792d3a740bd128222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1c17b4172c4ba792d3a740bd128222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34541 } } }
I20260620 15:21:17.346647 20840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:17.351738 20841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c1c17b4172c4ba792d3a740bd128222. Latest consensus state: current_term: 1 leader_uuid: "6c1c17b4172c4ba792d3a740bd128222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1c17b4172c4ba792d3a740bd128222" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34541 } } }
I20260620 15:21:17.352880 20841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:17.357102 20844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:17.358178 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:17.338941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39063 (local address 127.0.0.1:34541)
0620 15:21:17.339128 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:17.339136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:17.339587 (+ 451us) server_negotiation.cc:407] Connection header received
0620 15:21:17.339692 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:17.339697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:17.339782 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:17.339886 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:17.341501 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:17.342859 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:17.344858 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:17.345621 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:17.349412 (+ 3791us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:17.349783 (+ 371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:17.349792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:17.349800 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:17.349841 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:17.354435 (+ 4594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:17.355627 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:17.355632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:17.355637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:17.355808 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:17.356517 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:17.356524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:17.356529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:17.356925 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:17.357203 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:17.357579 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 15:21:17.357755 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:21:17.363272 20844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:17.374038 20844 catalog_manager.cc:1347] Generated new cluster ID: 6a9a6a0bcb88408da6d2453169ed32f9
I20260620 15:21:17.374164 20844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:17.399132 20844 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:17.401196 20844 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:17.409091 20844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c1c17b4172c4ba792d3a740bd128222: Generated new TSK 0
I20260620 15:21:17.409895 20844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:17.416395 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 20762
I20260620 15:21:17.417997 20824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:17.510582 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 20762
W20260620 15:21:17.527565 20747 connection.cc:535] client connection to 127.0.0.1:34541 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34541): Connection reset by peer (error 104)
2026-06-20T15:21:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:17 dist-test-slave-7s64 krb5kdc[20861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:17 dist-test-slave-7s64 krb5kdc[20861](info): set up 2 sockets
Jun 20 15:21:17 dist-test-slave-7s64 krb5kdc[20861](info): commencing operation
krb5kdc: starting...
W20260620 15:21:18.596778 389 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:18 dist-test-slave-7s64 krb5kdc[20861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:18.816746 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46415
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:19.128993 20874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:19.129287 20874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:19.129456 20874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:19.138793 20874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:19.138921 20874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:19.139001 20874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:19.139125 20874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:19.139192 20874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:19.150134 20874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:19.151348 20874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:19.152662 20874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:19.163724 20880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:19.163775 20882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:19.164383 20879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:19.164654 20874 server_base.cc:1034] running on GCE node
I20260620 15:21:19.165401 20874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:19.166695 20874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:19.167891 20874 hybrid_clock.cc:648] HybridClock initialized: now 1781968879167839 us; error 47 us; skew 500 ppm
I20260620 15:21:19.168237 20874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:19 dist-test-slave-7s64 krb5kdc[20861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:19.176344 20874 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:19.177887 20874 webserver.cc:460] Webserver started at http://127.0.0.1:41249/ using document root <none> and password file <none>
I20260620 15:21:19.178519 20874 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:19.178632 20874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:19.178893 20874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:19.181881 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "e5fe6b9036be47e997d03c47432cc9f5"
format_stamp: "Formatted at 2026-06-20 15:21:19 on dist-test-slave-7s64"
server_key: "278734cb3b0d58f37afa0a3e0a976849"
server_key_iv: "87973a3c8794a07da38421e52738616d"
server_key_version: "encryptionkey@0"
I20260620 15:21:19.182617 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "e5fe6b9036be47e997d03c47432cc9f5"
format_stamp: "Formatted at 2026-06-20 15:21:19 on dist-test-slave-7s64"
server_key: "278734cb3b0d58f37afa0a3e0a976849"
server_key_iv: "87973a3c8794a07da38421e52738616d"
server_key_version: "encryptionkey@0"
I20260620 15:21:19.187633 20874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 15:21:19.191011 20889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:19.192304 20874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:21:19.192552 20874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "e5fe6b9036be47e997d03c47432cc9f5"
format_stamp: "Formatted at 2026-06-20 15:21:19 on dist-test-slave-7s64"
server_key: "278734cb3b0d58f37afa0a3e0a976849"
server_key_iv: "87973a3c8794a07da38421e52738616d"
server_key_version: "encryptionkey@0"
I20260620 15:21:19.192741 20874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:19.204026 20874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:19.207818 20874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:19.208084 20874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:19.226382 20874 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46415
I20260620 15:21:19.226457 20940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46415 every 8 connection(s)
I20260620 15:21:19.227623 20874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:19.231220 20941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:19.235441 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 20874
I20260620 15:21:19.235635 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:19.236089 389 external_mini_cluster.cc:1428] Setting key 0dad1ee1112772d950d0201420bd4263
I20260620 15:21:19.243901 20941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5: Bootstrap starting.
I20260620 15:21:19.247393 20941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:19.248603 20941 log.cc:824] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:19.249215 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:19.237680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41659 (local address 127.0.0.1:46415)
0620 15:21:19.238416 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:19.238427 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:19.238467 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:21:19.240030 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:19.240051 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:19.240443 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:19.240808 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:19.241453 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:19.242771 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:19.244666 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:19.245326 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:19.245459 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:19.246833 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:19.246842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:19.246854 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:19.247351 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:19.247422 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:19.247445 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:19.247674 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:19.247897 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 15:21:19.248084 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":351,"threads_started":1}
I20260620 15:21:19.251240 20941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5: No bootstrap required, opened a new log
I20260620 15:21:19.252720 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:19.259043 20941 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5fe6b9036be47e997d03c47432cc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46415 } }
I20260620 15:21:19.259451 20941 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:19.259554 20941 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5fe6b9036be47e997d03c47432cc9f5, State: Initialized, Role: FOLLOWER
I20260620 15:21:19.260169 20941 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5fe6b9036be47e997d03c47432cc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46415 } }
I20260620 15:21:19.260454 20941 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:19.260608 20941 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:19.260772 20941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:19.263605 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:19.256875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45595 (local address 127.0.0.1:46415)
0620 15:21:19.257051 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:19.257057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:19.257583 (+ 526us) server_negotiation.cc:407] Connection header received
0620 15:21:19.257768 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:19.257771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:19.257826 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:19.257905 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:19.258478 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:19.259161 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:19.261067 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:19.261511 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:19.261608 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:19.262754 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:19.262760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:19.262763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:19.262778 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:19.262828 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:19.262842 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:19.262921 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:19.263220 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 15:21:19.263279 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:21:19.263749 20941 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5fe6b9036be47e997d03c47432cc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46415 } }
I20260620 15:21:19.264341 20941 leader_election.cc:302] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5fe6b9036be47e997d03c47432cc9f5; no voters:
I20260620 15:21:19.265017 20941 leader_election.cc:288] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:19.265118 20951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:19.265776 20951 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [term 1 LEADER]: Becoming Leader. State: Replica: e5fe6b9036be47e997d03c47432cc9f5, State: Running, Role: LEADER
I20260620 15:21:19.266357 20951 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5fe6b9036be47e997d03c47432cc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46415 } }
I20260620 15:21:19.267120 20941 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:19.269616 20953 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5fe6b9036be47e997d03c47432cc9f5. Latest consensus state: current_term: 1 leader_uuid: "e5fe6b9036be47e997d03c47432cc9f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5fe6b9036be47e997d03c47432cc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46415 } } }
I20260620 15:21:19.270035 20953 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:19.269608 20952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5fe6b9036be47e997d03c47432cc9f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5fe6b9036be47e997d03c47432cc9f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46415 } } }
I20260620 15:21:19.270473 20952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:19.271638 20957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:19.277678 20957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:19.286386 20957 catalog_manager.cc:1347] Generated new cluster ID: f5b87d4336da472092c6f90da1194ff6
I20260620 15:21:19.286496 20957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:19.312656 20957 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:19.314482 20957 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:19.322592 20957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5fe6b9036be47e997d03c47432cc9f5: Generated new TSK 0
I20260620 15:21:19.323604 20957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:19.337772 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 20874
W20260620 15:21:19.338876 20936 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41659
W20260620 15:21:19.339632 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 20874: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:19.339828 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 20874
W20260620 15:21:19.361536 20859 connection.cc:535] client connection to 127.0.0.1:46415 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46415): Connection reset by peer (error 104)
2026-06-20T15:21:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1832 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:19 dist-test-slave-7s64 krb5kdc[20972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:19 dist-test-slave-7s64 krb5kdc[20972](info): set up 2 sockets
Jun 20 15:21:19 dist-test-slave-7s64 krb5kdc[20972](info): commencing operation
krb5kdc: starting...
W20260620 15:21:20.428853 389 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:20 dist-test-slave-7s64 krb5kdc[20972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:20.647423 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:20.964327 20985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:20.964640 20985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:20.964809 20985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:20.971671 20985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:20.971810 20985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:20.971916 20985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:20.972024 20985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:20.972112 20985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:20.981382 20985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:20.982564 20985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:20.983835 20985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:20.994974 20990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:20.995081 20993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:20.995081 20991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:20.995968 20985 server_base.cc:1034] running on GCE node
I20260620 15:21:20.996729 20985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:20.997998 20985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:20.999195 20985 hybrid_clock.cc:648] HybridClock initialized: now 1781968880999154 us; error 32 us; skew 500 ppm
I20260620 15:21:20.999516 20985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:21 dist-test-slave-7s64 krb5kdc[20972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:21.006037 20985 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:21.007436 20985 webserver.cc:460] Webserver started at http://127.0.0.1:41579/ using document root <none> and password file <none>
I20260620 15:21:21.008046 20985 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:21.008169 20985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:21.008553 20985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:21.011317 20985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "7140293767d3481e9d97918d99f7ab01"
format_stamp: "Formatted at 2026-06-20 15:21:21 on dist-test-slave-7s64"
server_key: "974a815965c174966c44b4ca73023f7a"
server_key_iv: "46b82e4402acce64142f47892375c930"
server_key_version: "encryptionkey@0"
I20260620 15:21:21.012043 20985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "7140293767d3481e9d97918d99f7ab01"
format_stamp: "Formatted at 2026-06-20 15:21:21 on dist-test-slave-7s64"
server_key: "974a815965c174966c44b4ca73023f7a"
server_key_iv: "46b82e4402acce64142f47892375c930"
server_key_version: "encryptionkey@0"
I20260620 15:21:21.016942 20985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 15:21:21.020082 21000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:21.021381 20985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:21:21.021560 20985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "7140293767d3481e9d97918d99f7ab01"
format_stamp: "Formatted at 2026-06-20 15:21:21 on dist-test-slave-7s64"
server_key: "974a815965c174966c44b4ca73023f7a"
server_key_iv: "46b82e4402acce64142f47892375c930"
server_key_version: "encryptionkey@0"
I20260620 15:21:21.021754 20985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:21.027221 20985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:21.031374 20985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:21.031659 20985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:21.049499 20985 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39979
I20260620 15:21:21.049573 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39979 every 8 connection(s)
I20260620 15:21:21.050761 20985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:21.054231 21052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:21.056841 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 20985
I20260620 15:21:21.057020 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:21.057544 389 external_mini_cluster.cc:1428] Setting key bd60ab734feb5ebc466e9ee059281550
I20260620 15:21:21.064831 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:21.059219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59347 (local address 127.0.0.1:39979)
0620 15:21:21.059981 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:21.059993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:21.060031 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:21:21.061764 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:21.061786 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:21.062091 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:21.062391 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:21.063353 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:21.063359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:21.063371 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:21.063384 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:21.063438 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:21.063457 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:21.063624 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:21.063743 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 15:21:21.063933 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":368,"threads_started":1}
I20260620 15:21:21.065892 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01: Bootstrap starting.
I20260620 15:21:21.068128 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:21.068996 21052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:21.070119 21052 log.cc:824] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:21.073105 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01: No bootstrap required, opened a new log
I20260620 15:21:21.075368 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:21.073104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53555 (local address 127.0.0.1:39979)
0620 15:21:21.073270 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:21.073276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:21.073684 (+ 408us) server_negotiation.cc:407] Connection header received
0620 15:21:21.073787 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:21.073794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:21.073857 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:21.073959 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:21.074768 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:21.074778 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:21.074783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:21.074801 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:21.074853 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:21.074871 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:21.074992 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:21.075043 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 15:21:21.075093 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:21:21.081419 21052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7140293767d3481e9d97918d99f7ab01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39979 } }
I20260620 15:21:21.081754 21052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:21.081866 21052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7140293767d3481e9d97918d99f7ab01, State: Initialized, Role: FOLLOWER
I20260620 15:21:21.082391 21052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7140293767d3481e9d97918d99f7ab01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39979 } }
I20260620 15:21:21.082584 21052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:21.082737 21052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:21.082902 21052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:21.084975 21052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7140293767d3481e9d97918d99f7ab01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39979 } }
I20260620 15:21:21.085506 21052 leader_election.cc:302] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7140293767d3481e9d97918d99f7ab01; no voters:
I20260620 15:21:21.086030 21052 leader_election.cc:288] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:21.086164 21062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:21.087347 21062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [term 1 LEADER]: Becoming Leader. State: Replica: 7140293767d3481e9d97918d99f7ab01, State: Running, Role: LEADER
I20260620 15:21:21.087800 21062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7140293767d3481e9d97918d99f7ab01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39979 } }
I20260620 15:21:21.087931 21052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:21.090584 21064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7140293767d3481e9d97918d99f7ab01. Latest consensus state: current_term: 1 leader_uuid: "7140293767d3481e9d97918d99f7ab01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7140293767d3481e9d97918d99f7ab01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39979 } } }
I20260620 15:21:21.090969 21064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:21.091454 21063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7140293767d3481e9d97918d99f7ab01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7140293767d3481e9d97918d99f7ab01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39979 } } }
I20260620 15:21:21.091735 21063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:21.092581 21071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:21.097147 21071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:21.105057 21071 catalog_manager.cc:1347] Generated new cluster ID: 837ab66398154fec9e037e9c5ed08885
I20260620 15:21:21.105154 21071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:21.121963 21071 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:21.122364 21071 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:21.131937 21071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7140293767d3481e9d97918d99f7ab01: Generated new TSK 0
I20260620 15:21:21.132859 21071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:21.147586 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 20985
W20260620 15:21:21.148658 21047 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59347
W20260620 15:21:21.149269 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 20985: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:21.149411 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 20985
2026-06-20T15:21:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1804 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:21 dist-test-slave-7s64 krb5kdc[21083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:21 dist-test-slave-7s64 krb5kdc[21083](info): set up 2 sockets
Jun 20 15:21:21 dist-test-slave-7s64 krb5kdc[21083](info): commencing operation
krb5kdc: starting...
W20260620 15:21:22.236873 389 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:22 dist-test-slave-7s64 krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:22.479668 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:22.799856 21096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:22.800127 21096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:22.800288 21096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:22.807457 21096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:22.807561 21096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:22.807664 21096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:22.807777 21096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:22.807857 21096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:22.817543 21096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:22.818804 21096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:22.820155 21096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:22.831753 21102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:22.831815 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:22.832106 21101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:22.832845 21096 server_base.cc:1034] running on GCE node
I20260620 15:21:22.833552 21096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:22.834856 21096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:22.836093 21096 hybrid_clock.cc:648] HybridClock initialized: now 1781968882836055 us; error 53 us; skew 500 ppm
I20260620 15:21:22.836464 21096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:22 dist-test-slave-7s64 krb5kdc[21083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:22.839591 21096 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:22.841392 21096 webserver.cc:460] Webserver started at http://127.0.0.1:36695/ using document root <none> and password file <none>
I20260620 15:21:22.842059 21096 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:22.842204 21096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:22.842509 21096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:22.845495 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "8292dc978bab4c19bbc7cd94e0b1be22"
format_stamp: "Formatted at 2026-06-20 15:21:22 on dist-test-slave-7s64"
server_key: "d6147b3c752b82aecdaefa0e25160131"
server_key_iv: "5b8ea1dcc2e52b03d4e1069227ce4c8b"
server_key_version: "encryptionkey@0"
I20260620 15:21:22.846246 21096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "8292dc978bab4c19bbc7cd94e0b1be22"
format_stamp: "Formatted at 2026-06-20 15:21:22 on dist-test-slave-7s64"
server_key: "d6147b3c752b82aecdaefa0e25160131"
server_key_iv: "5b8ea1dcc2e52b03d4e1069227ce4c8b"
server_key_version: "encryptionkey@0"
I20260620 15:21:22.851136 21096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:21:22.854395 21111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:22.856040 21096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:21:22.856235 21096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "8292dc978bab4c19bbc7cd94e0b1be22"
format_stamp: "Formatted at 2026-06-20 15:21:22 on dist-test-slave-7s64"
server_key: "d6147b3c752b82aecdaefa0e25160131"
server_key_iv: "5b8ea1dcc2e52b03d4e1069227ce4c8b"
server_key_version: "encryptionkey@0"
I20260620 15:21:22.856431 21096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:22.882001 21096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:22.886039 21096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:22.886328 21096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:22.904729 21096 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36307
I20260620 15:21:22.904767 21162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36307 every 8 connection(s)
I20260620 15:21:22.906060 21096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:22.910285 21163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:22.911428 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 21096
I20260620 15:21:22.911620 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:22.912082 389 external_mini_cluster.cc:1428] Setting key fc3e51165f01a884e784d0240f3c2b1b
I20260620 15:21:22.924315 21163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22: Bootstrap starting.
Jun 20 15:21:22 dist-test-slave-7s64 krb5kdc[21083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:21:22.927271 21163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:22.928625 21163 log.cc:824] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:22.931950 21163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22: No bootstrap required, opened a new log
I20260620 15:21:22.935873 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:22.913899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37383 (local address 127.0.0.1:36307)
0620 15:21:22.914692 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:22.914712 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:22.914775 (+ 63us) server_negotiation.cc:407] Connection header received
0620 15:21:22.916820 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:22.916848 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:22.917360 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:22.917815 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:22.919018 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:22.921065 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:22.922297 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:22.923304 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:22.923484 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:22.927476 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:22.927486 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:22.927503 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:22.927551 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:22.931580 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:22.932321 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:22.932334 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:22.932344 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:22.932578 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:22.932939 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:22.932943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:22.932947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:22.933576 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:22.933880 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:22.934167 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 15:21:22.934576 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":304,"threads_started":1}
I20260620 15:21:22.939502 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:22.941263 21163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292dc978bab4c19bbc7cd94e0b1be22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } }
I20260620 15:21:22.941586 21163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:22.941676 21163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8292dc978bab4c19bbc7cd94e0b1be22, State: Initialized, Role: FOLLOWER
I20260620 15:21:22.942276 21163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292dc978bab4c19bbc7cd94e0b1be22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } }
I20260620 15:21:22.942546 21163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:22.942687 21163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:22.942859 21163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:22.945390 21163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292dc978bab4c19bbc7cd94e0b1be22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } }
I20260620 15:21:22.945835 21163 leader_election.cc:302] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8292dc978bab4c19bbc7cd94e0b1be22; no voters:
I20260620 15:21:22.946481 21163 leader_election.cc:288] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:22.946553 21173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:22.948406 21163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:22.952800 21173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [term 1 LEADER]: Becoming Leader. State: Replica: 8292dc978bab4c19bbc7cd94e0b1be22, State: Running, Role: LEADER
I20260620 15:21:22.953298 21173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292dc978bab4c19bbc7cd94e0b1be22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } }
I20260620 15:21:22.955884 21174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8292dc978bab4c19bbc7cd94e0b1be22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292dc978bab4c19bbc7cd94e0b1be22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } } }
I20260620 15:21:22.956343 21174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:22.956954 21175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8292dc978bab4c19bbc7cd94e0b1be22. Latest consensus state: current_term: 1 leader_uuid: "8292dc978bab4c19bbc7cd94e0b1be22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8292dc978bab4c19bbc7cd94e0b1be22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36307 } } }
I20260620 15:21:22.957304 21175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:22.961879 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:22.943874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40381 (local address 127.0.0.1:36307)
0620 15:21:22.944040 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:22.944046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:22.944393 (+ 347us) server_negotiation.cc:407] Connection header received
0620 15:21:22.944509 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:22.944513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:22.944583 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:22.944678 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:22.946154 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:22.947406 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:22.949182 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:22.950125 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:22.950317 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:22.954038 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:22.954045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:22.954049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:22.954083 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:22.958563 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:22.959642 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:22.959650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:22.959656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:22.959818 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:22.960346 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:22.960349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:22.960351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:22.960761 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:22.961028 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:22.961316 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 15:21:22.961515 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:21:22.962008 21180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:22.967423 21180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:22.976141 21180 catalog_manager.cc:1347] Generated new cluster ID: bc1beef1f77e4049828f4761d38d4ed1
I20260620 15:21:22.976267 21180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:22.990484 21180 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:22.991629 21180 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:22.999851 21180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8292dc978bab4c19bbc7cd94e0b1be22: Generated new TSK 0
I20260620 15:21:23.000600 21180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:23.010970 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21096
I20260620 15:21:23.011961 21158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:23.104472 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 21096
2026-06-20T15:21:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:23 dist-test-slave-7s64 krb5kdc[21195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:23 dist-test-slave-7s64 krb5kdc[21195](info): set up 2 sockets
Jun 20 15:21:23 dist-test-slave-7s64 krb5kdc[21195](info): commencing operation
krb5kdc: starting...
W20260620 15:21:24.188886 389 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:24 dist-test-slave-7s64 krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:24.409015 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37947
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:24.734823 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:24.735078 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:24.735240 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:24.742295 21208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:24.742399 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:24.742520 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:24.742624 21208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:24.742712 21208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:24.752187 21208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:24.753348 21208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:24.754595 21208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:24.766289 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:24.766220 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:24.766322 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:24.767309 21208 server_base.cc:1034] running on GCE node
I20260620 15:21:24.768011 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:24.769351 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:24.770629 21208 hybrid_clock.cc:648] HybridClock initialized: now 1781968884770573 us; error 49 us; skew 500 ppm
I20260620 15:21:24.770983 21208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:24 dist-test-slave-7s64 krb5kdc[21195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:24.774327 21208 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:24.775658 21208 webserver.cc:460] Webserver started at http://127.0.0.1:42097/ using document root <none> and password file <none>
I20260620 15:21:24.776284 21208 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:24.776425 21208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:24.776736 21208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:24.779469 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "6c9978c8d67b4175850272960b627cc1"
format_stamp: "Formatted at 2026-06-20 15:21:24 on dist-test-slave-7s64"
server_key: "cf3e548acc83f8a1a50acdb11403f06f"
server_key_iv: "79fe8b064308f7472a9573ad2d4963d1"
server_key_version: "encryptionkey@0"
I20260620 15:21:24.780169 21208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "6c9978c8d67b4175850272960b627cc1"
format_stamp: "Formatted at 2026-06-20 15:21:24 on dist-test-slave-7s64"
server_key: "cf3e548acc83f8a1a50acdb11403f06f"
server_key_iv: "79fe8b064308f7472a9573ad2d4963d1"
server_key_version: "encryptionkey@0"
I20260620 15:21:24.784843 21208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:21:24.787909 21223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:24.789160 21208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:21:24.789338 21208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "6c9978c8d67b4175850272960b627cc1"
format_stamp: "Formatted at 2026-06-20 15:21:24 on dist-test-slave-7s64"
server_key: "cf3e548acc83f8a1a50acdb11403f06f"
server_key_iv: "79fe8b064308f7472a9573ad2d4963d1"
server_key_version: "encryptionkey@0"
I20260620 15:21:24.789515 21208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:24.805989 21208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:24.809782 21208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:24.810045 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:24.827994 21208 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37947
I20260620 15:21:24.828073 21274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37947 every 8 connection(s)
I20260620 15:21:24.829272 21208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:24.832901 21275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:24.838572 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 21208
I20260620 15:21:24.838757 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:24.839184 389 external_mini_cluster.cc:1428] Setting key e5147ea0e6a9d28b8f20e79b3e29da45
I20260620 15:21:24.843781 21275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1: Bootstrap starting.
I20260620 15:21:24.847558 21275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:24.848582 21275 log.cc:824] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:24.851426 21275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1: No bootstrap required, opened a new log
I20260620 15:21:24.853868 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:24.840996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56165 (local address 127.0.0.1:37947)
0620 15:21:24.841810 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:24.841824 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:24.841870 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:21:24.844186 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:24.844211 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:24.844720 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:24.845132 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:24.845652 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:24.847560 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:24.849009 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:24.849851 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:24.850046 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:24.851142 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:24.851152 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:24.851168 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:24.851897 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:24.851992 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:24.852029 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:24.852146 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:24.852376 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 15:21:24.852631 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":331,"threads_started":1}
I20260620 15:21:24.857487 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:24.860277 21275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9978c8d67b4175850272960b627cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260620 15:21:24.860683 21275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:24.860808 21275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9978c8d67b4175850272960b627cc1, State: Initialized, Role: FOLLOWER
I20260620 15:21:24.861320 21275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9978c8d67b4175850272960b627cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260620 15:21:24.861553 21275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:24.861723 21275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:24.861918 21275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:24.864838 21275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9978c8d67b4175850272960b627cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260620 15:21:24.865290 21275 leader_election.cc:302] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c9978c8d67b4175850272960b627cc1; no voters:
I20260620 15:21:24.865909 21275 leader_election.cc:288] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:24.866010 21285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:24.867897 21285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [term 1 LEADER]: Becoming Leader. State: Replica: 6c9978c8d67b4175850272960b627cc1, State: Running, Role: LEADER
I20260620 15:21:24.868055 21275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:24.868549 21285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9978c8d67b4175850272960b627cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } }
I20260620 15:21:24.868906 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:24.861789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37583 (local address 127.0.0.1:37947)
0620 15:21:24.861962 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:24.861967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:24.862354 (+ 387us) server_negotiation.cc:407] Connection header received
0620 15:21:24.862456 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:24.862461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:24.862535 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:24.862631 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:24.863063 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:24.864144 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:24.865568 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:24.866477 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:24.866627 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:24.868079 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:24.868086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:24.868091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:24.868110 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:24.868156 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:24.868180 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:24.868275 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:24.868453 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 15:21:24.868570 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 15:21:24.872458 21287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c9978c8d67b4175850272960b627cc1. Latest consensus state: current_term: 1 leader_uuid: "6c9978c8d67b4175850272960b627cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9978c8d67b4175850272960b627cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } } }
I20260620 15:21:24.872511 21286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c9978c8d67b4175850272960b627cc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c9978c8d67b4175850272960b627cc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37947 } } }
I20260620 15:21:24.872923 21287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:24.872923 21286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:24.873792 21297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:24.879264 21297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:24.887430 21297 catalog_manager.cc:1347] Generated new cluster ID: edc0e91971b64ec2b770ee2314fa9a66
I20260620 15:21:24.887547 21297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:24.907155 21297 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:24.908380 21297 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:24.920051 21297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c9978c8d67b4175850272960b627cc1: Generated new TSK 0
I20260620 15:21:24.920876 21297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:24.926452 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21208
W20260620 15:21:24.927524 21270 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56165
W20260620 15:21:24.928239 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21208: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:24.928413 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 21208
2026-06-20T15:21:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1823 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:25 dist-test-slave-7s64 krb5kdc[21306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:25 dist-test-slave-7s64 krb5kdc[21306](info): set up 2 sockets
Jun 20 15:21:25 dist-test-slave-7s64 krb5kdc[21306](info): commencing operation
krb5kdc: starting...
W20260620 15:21:26.020853 389 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:26 dist-test-slave-7s64 krb5kdc[21306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:26.237576 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:26.551614 21319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:26.551859 21319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:26.552030 21319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:26.559350 21319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:26.559453 21319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:26.559545 21319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:26.559660 21319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:26.559742 21319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:26.569412 21319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:26.570557 21319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:26.571796 21319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:26.583114 21327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:26.583241 21324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:26.583148 21325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:26.583837 21319 server_base.cc:1034] running on GCE node
I20260620 15:21:26.584620 21319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:26.585880 21319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:26.587070 21319 hybrid_clock.cc:648] HybridClock initialized: now 1781968886587036 us; error 33 us; skew 500 ppm
I20260620 15:21:26.587410 21319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:26 dist-test-slave-7s64 krb5kdc[21306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:26.598618 21319 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:26.600172 21319 webserver.cc:460] Webserver started at http://127.0.0.1:37835/ using document root <none> and password file <none>
I20260620 15:21:26.600939 21319 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:26.601082 21319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:26.601383 21319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:26.604254 21319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "d2c906f06f4144a7840b1537468b13a3"
format_stamp: "Formatted at 2026-06-20 15:21:26 on dist-test-slave-7s64"
server_key: "b6805da4e4567801c609a6bdb2867ddd"
server_key_iv: "caefa2b5e23dbd576ac2c8b9f7ac833c"
server_key_version: "encryptionkey@0"
I20260620 15:21:26.605072 21319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "d2c906f06f4144a7840b1537468b13a3"
format_stamp: "Formatted at 2026-06-20 15:21:26 on dist-test-slave-7s64"
server_key: "b6805da4e4567801c609a6bdb2867ddd"
server_key_iv: "caefa2b5e23dbd576ac2c8b9f7ac833c"
server_key_version: "encryptionkey@0"
I20260620 15:21:26.609941 21319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 15:21:26.613225 21334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:26.614428 21319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:21:26.614600 21319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "d2c906f06f4144a7840b1537468b13a3"
format_stamp: "Formatted at 2026-06-20 15:21:26 on dist-test-slave-7s64"
server_key: "b6805da4e4567801c609a6bdb2867ddd"
server_key_iv: "caefa2b5e23dbd576ac2c8b9f7ac833c"
server_key_version: "encryptionkey@0"
I20260620 15:21:26.614809 21319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:26.620244 21319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:26.624290 21319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:26.624657 21319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:26.642208 21319 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45693
I20260620 15:21:26.642277 21385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45693 every 8 connection(s)
I20260620 15:21:26.643357 21319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:26.646183 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 21319
I20260620 15:21:26.646348 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:26.646900 389 external_mini_cluster.cc:1428] Setting key 9caa778ece7c522bec238c9798ac57f7
I20260620 15:21:26.647099 21386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:26.654247 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:26.648450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50815 (local address 127.0.0.1:45693)
0620 15:21:26.649300 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:26.649310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:26.649347 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:21:26.651281 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:26.651299 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:26.651585 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:26.651876 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:26.652849 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:26.652855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:26.652869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:26.652882 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:26.652933 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:26.652950 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:26.653110 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:26.653215 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 15:21:26.653412 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":366,"threads_started":1}
I20260620 15:21:26.657549 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:26.659603 21386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3: Bootstrap starting.
I20260620 15:21:26.663002 21386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:26.663961 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:26.661586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46859 (local address 127.0.0.1:45693)
0620 15:21:26.661762 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:26.661769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:26.662052 (+ 283us) server_negotiation.cc:407] Connection header received
0620 15:21:26.662261 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:26.662265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:26.662303 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:26.662402 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:26.663261 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:26.663269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:26.663273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:26.663294 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:26.663327 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:26.663343 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:26.663459 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:26.663636 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 15:21:26.663695 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:21:26.664251 21386 log.cc:824] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:26.666829 21386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3: No bootstrap required, opened a new log
I20260620 15:21:26.674136 21386 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c906f06f4144a7840b1537468b13a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260620 15:21:26.674546 21386 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:26.674639 21386 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c906f06f4144a7840b1537468b13a3, State: Initialized, Role: FOLLOWER
I20260620 15:21:26.675086 21386 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c906f06f4144a7840b1537468b13a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260620 15:21:26.675275 21386 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:26.675381 21386 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:26.675491 21386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:26.677472 21386 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c906f06f4144a7840b1537468b13a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260620 15:21:26.677907 21386 leader_election.cc:302] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c906f06f4144a7840b1537468b13a3; no voters:
I20260620 15:21:26.678364 21386 leader_election.cc:288] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:26.678498 21396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:26.679694 21396 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [term 1 LEADER]: Becoming Leader. State: Replica: d2c906f06f4144a7840b1537468b13a3, State: Running, Role: LEADER
I20260620 15:21:26.680174 21396 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c906f06f4144a7840b1537468b13a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } }
I20260620 15:21:26.680302 21386 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:26.683096 21397 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2c906f06f4144a7840b1537468b13a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c906f06f4144a7840b1537468b13a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } } }
I20260620 15:21:26.683540 21397 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:26.683071 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2c906f06f4144a7840b1537468b13a3. Latest consensus state: current_term: 1 leader_uuid: "d2c906f06f4144a7840b1537468b13a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c906f06f4144a7840b1537468b13a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45693 } } }
I20260620 15:21:26.684052 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:26.685547 21405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:26.689882 21405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:26.697630 21405 catalog_manager.cc:1347] Generated new cluster ID: 06cf6f82a3a74de489ee97215f3e6774
I20260620 15:21:26.697742 21405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:26.711937 21405 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:26.712303 21405 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:26.719107 21405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2c906f06f4144a7840b1537468b13a3: Generated new TSK 0
I20260620 15:21:26.719828 21405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:26.730576 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21319
W20260620 15:21:26.732120 21381 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50815
W20260620 15:21:26.732767 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21319: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:26.732924 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 21319
2026-06-20T15:21:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1806 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:26 dist-test-slave-7s64 krb5kdc[21417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:26 dist-test-slave-7s64 krb5kdc[21417](info): set up 2 sockets
Jun 20 15:21:26 dist-test-slave-7s64 krb5kdc[21417](info): commencing operation
krb5kdc: starting...
W20260620 15:21:27.852878 389 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:28 dist-test-slave-7s64 krb5kdc[21417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:28.077947 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:28.389056 21430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:28.389355 21430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:28.389521 21430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:28.396390 21430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:28.396544 21430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:28.396682 21430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:28.396796 21430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:28.396885 21430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:28.406105 21430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:28.407230 21430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:28.408427 21430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:28.419629 21438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:28.419683 21436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:28.419718 21435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:28.420806 21430 server_base.cc:1034] running on GCE node
I20260620 15:21:28.421516 21430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:28.422780 21430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:28.423995 21430 hybrid_clock.cc:648] HybridClock initialized: now 1781968888423960 us; error 44 us; skew 500 ppm
I20260620 15:21:28.424324 21430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:28 dist-test-slave-7s64 krb5kdc[21417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:28.427299 21430 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:28.428696 21430 webserver.cc:460] Webserver started at http://127.0.0.1:35629/ using document root <none> and password file <none>
I20260620 15:21:28.429325 21430 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:28.429469 21430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:28.429790 21430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:28.432611 21430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "e7ff84088036472d8aba590bfc3eb973"
format_stamp: "Formatted at 2026-06-20 15:21:28 on dist-test-slave-7s64"
server_key: "5584939d5e2d69c4adff360cd3a17706"
server_key_iv: "381f2b279ef42b33bcbc0ac31a78a078"
server_key_version: "encryptionkey@0"
I20260620 15:21:28.433351 21430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "e7ff84088036472d8aba590bfc3eb973"
format_stamp: "Formatted at 2026-06-20 15:21:28 on dist-test-slave-7s64"
server_key: "5584939d5e2d69c4adff360cd3a17706"
server_key_iv: "381f2b279ef42b33bcbc0ac31a78a078"
server_key_version: "encryptionkey@0"
I20260620 15:21:28.438010 21430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:21:28.441206 21445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:28.442340 21430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:21:28.442510 21430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "e7ff84088036472d8aba590bfc3eb973"
format_stamp: "Formatted at 2026-06-20 15:21:28 on dist-test-slave-7s64"
server_key: "5584939d5e2d69c4adff360cd3a17706"
server_key_iv: "381f2b279ef42b33bcbc0ac31a78a078"
server_key_version: "encryptionkey@0"
I20260620 15:21:28.442690 21430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:28.461697 21430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:28.465612 21430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:28.465883 21430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:28.484203 21430 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40667
I20260620 15:21:28.484241 21496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40667 every 8 connection(s)
I20260620 15:21:28.485540 21430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:28.485911 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 21430
I20260620 15:21:28.486075 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:28.486685 389 external_mini_cluster.cc:1428] Setting key 7faeb9b7740743ee87d51c26f98b5d2c
I20260620 15:21:28.489706 21497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:21:28 dist-test-slave-7s64 krb5kdc[21417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:21:28.502621 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973: Bootstrap starting.
I20260620 15:21:28.505576 21497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:28.506783 21497 log.cc:824] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:28.509914 21497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973: No bootstrap required, opened a new log
I20260620 15:21:28.510524 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:28.488407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:56667 (local address 127.0.0.1:40667)
0620 15:21:28.489216 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:28.489227 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:28.489265 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:21:28.490853 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:28.490874 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:28.491235 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:28.491521 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:28.493341 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:28.494632 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:28.496587 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:28.497345 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:28.497493 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:28.502459 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:28.502467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:28.502479 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:28.502513 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:28.505931 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:28.506843 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:28.506850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:28.506855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:28.507067 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:28.507629 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:28.507637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:28.507640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:28.508335 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:28.508650 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:28.509131 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 15:21:28.509429 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":366,"threads_started":1}
I20260620 15:21:28.514441 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:28.519802 21497 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff84088036472d8aba590bfc3eb973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } }
I20260620 15:21:28.520123 21497 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:28.520226 21497 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ff84088036472d8aba590bfc3eb973, State: Initialized, Role: FOLLOWER
I20260620 15:21:28.520850 21497 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff84088036472d8aba590bfc3eb973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } }
I20260620 15:21:28.521090 21497 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:28.521227 21497 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:28.521406 21497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:28.523516 21497 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff84088036472d8aba590bfc3eb973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } }
I20260620 15:21:28.523988 21497 leader_election.cc:302] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7ff84088036472d8aba590bfc3eb973; no voters:
I20260620 15:21:28.524672 21497 leader_election.cc:288] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:28.524783 21507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:28.526813 21497 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:28.528828 21507 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [term 1 LEADER]: Becoming Leader. State: Replica: e7ff84088036472d8aba590bfc3eb973, State: Running, Role: LEADER
I20260620 15:21:28.529557 21507 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff84088036472d8aba590bfc3eb973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } }
I20260620 15:21:28.532927 21509 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7ff84088036472d8aba590bfc3eb973. Latest consensus state: current_term: 1 leader_uuid: "e7ff84088036472d8aba590bfc3eb973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff84088036472d8aba590bfc3eb973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } } }
I20260620 15:21:28.532996 21508 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7ff84088036472d8aba590bfc3eb973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ff84088036472d8aba590bfc3eb973" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40667 } } }
I20260620 15:21:28.533376 21509 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:28.533425 21508 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:28.538513 21500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:28.519388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:34125 (local address 127.0.0.1:40667)
0620 15:21:28.519576 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:28.519584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:28.520149 (+ 565us) server_negotiation.cc:407] Connection header received
0620 15:21:28.520337 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:28.520343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:28.520450 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:28.520585 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:28.522243 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:28.523082 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:28.524826 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:28.525636 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:28.525758 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:28.529313 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:28.529322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:28.529330 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:28.529371 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:28.534153 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:28.535616 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:28.535626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:28.535633 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:28.535890 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:28.536685 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:28.536694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:28.536700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:28.537130 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:28.537496 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:28.537874 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 15:21:28.538050 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:21:28.540619 21517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:28.544446 21517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:28.553644 21517 catalog_manager.cc:1347] Generated new cluster ID: 7643a80a415246d59c4d91c36d499ae2
I20260620 15:21:28.553753 21517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:28.583868 21517 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:28.585240 21517 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:28.592257 21517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7ff84088036472d8aba590bfc3eb973: Generated new TSK 0
I20260620 15:21:28.593058 21517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:28.606544 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21430
I20260620 15:21:28.608057 21492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:28.697672 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 21430
W20260620 15:21:28.714735 21415 connection.cc:535] client connection to 127.0.0.1:40667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40667): Connection reset by peer (error 104)
2026-06-20T15:21:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1963 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:28 dist-test-slave-7s64 krb5kdc[21529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:28 dist-test-slave-7s64 krb5kdc[21529](info): set up 2 sockets
Jun 20 15:21:28 dist-test-slave-7s64 krb5kdc[21529](info): commencing operation
krb5kdc: starting...
W20260620 15:21:29.804838 389 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:29 dist-test-slave-7s64 krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:30.025691 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:30.341080 21542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:30.341333 21542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:30.341419 21542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:30.348320 21542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:30.348414 21542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:30.348459 21542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:30.348578 21542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:30.348628 21542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:30.357913 21542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:30.358973 21542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:30.360157 21542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:30.371353 21547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:30.371485 21548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:30.372113 21550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:30.372726 21542 server_base.cc:1034] running on GCE node
I20260620 15:21:30.373420 21542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:30.374701 21542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:30.375929 21542 hybrid_clock.cc:648] HybridClock initialized: now 1781968890375878 us; error 39 us; skew 500 ppm
I20260620 15:21:30.376266 21542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:30 dist-test-slave-7s64 krb5kdc[21529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:30.379230 21542 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:30.380903 21542 webserver.cc:460] Webserver started at http://127.0.0.1:34415/ using document root <none> and password file <none>
I20260620 15:21:30.381700 21542 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:30.381803 21542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:30.382032 21542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:30.384932 21542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "12fe6862f41e4f30b31ebb3a3703e6dd"
format_stamp: "Formatted at 2026-06-20 15:21:30 on dist-test-slave-7s64"
server_key: "4e7640c5b302b4549271869f059dd1b4"
server_key_iv: "01426b82669a022561f99fe80bea1a0a"
server_key_version: "encryptionkey@0"
I20260620 15:21:30.385623 21542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "12fe6862f41e4f30b31ebb3a3703e6dd"
format_stamp: "Formatted at 2026-06-20 15:21:30 on dist-test-slave-7s64"
server_key: "4e7640c5b302b4549271869f059dd1b4"
server_key_iv: "01426b82669a022561f99fe80bea1a0a"
server_key_version: "encryptionkey@0"
I20260620 15:21:30.390254 21542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:21:30.393673 21557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:30.394912 21542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:21:30.395071 21542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "12fe6862f41e4f30b31ebb3a3703e6dd"
format_stamp: "Formatted at 2026-06-20 15:21:30 on dist-test-slave-7s64"
server_key: "4e7640c5b302b4549271869f059dd1b4"
server_key_iv: "01426b82669a022561f99fe80bea1a0a"
server_key_version: "encryptionkey@0"
I20260620 15:21:30.395260 21542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:30.421192 21542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:30.424932 21542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:30.425199 21542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:30.442893 21542 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39337
I20260620 15:21:30.442966 21608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39337 every 8 connection(s)
I20260620 15:21:30.444111 21542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:30.445869 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 21542
I20260620 15:21:30.446055 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:30.446461 389 external_mini_cluster.cc:1428] Setting key 645c6aef99289e7eb85bacb52fb7fb9e
I20260620 15:21:30.448036 21609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:30.459429 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:30.448148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:48513 (local address 127.0.0.1:39337)
0620 15:21:30.448918 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:30.448928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:30.448964 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:21:30.450521 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:30.450541 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:30.450891 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:30.451192 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:30.451764 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:30.453105 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:30.454978 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:30.455656 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:30.455792 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:30.457165 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:30.457172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:30.457185 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:30.457702 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:30.457776 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:30.457803 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:30.457925 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:30.458340 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 15:21:30.458521 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":339,"threads_started":1}
I20260620 15:21:30.460673 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd: Bootstrap starting.
I20260620 15:21:30.463235 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:30.463907 21609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:30.464893 21609 log.cc:824] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:30.467510 21609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd: No bootstrap required, opened a new log
I20260620 15:21:30.474290 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:30.467679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:49361 (local address 127.0.0.1:39337)
0620 15:21:30.467874 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:30.467883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:30.468298 (+ 415us) server_negotiation.cc:407] Connection header received
0620 15:21:30.468465 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:30.468469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:30.468579 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:30.468662 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:30.469220 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:30.469987 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:30.471820 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:30.472304 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:30.472398 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:30.473529 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:30.473533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:30.473536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:30.473553 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:30.473586 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:30.473602 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:30.473691 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:30.473899 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 15:21:30.473938 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:21:30.476732 21609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe6862f41e4f30b31ebb3a3703e6dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260620 15:21:30.477072 21609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:30.477166 21609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fe6862f41e4f30b31ebb3a3703e6dd, State: Initialized, Role: FOLLOWER
I20260620 15:21:30.477690 21609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe6862f41e4f30b31ebb3a3703e6dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260620 15:21:30.477934 21609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:30.478077 21609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:30.478252 21609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:30.480233 21609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe6862f41e4f30b31ebb3a3703e6dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260620 15:21:30.480829 21609 leader_election.cc:302] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12fe6862f41e4f30b31ebb3a3703e6dd; no voters:
I20260620 15:21:30.481405 21609 leader_election.cc:288] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:30.481565 21619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:30.482786 21619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [term 1 LEADER]: Becoming Leader. State: Replica: 12fe6862f41e4f30b31ebb3a3703e6dd, State: Running, Role: LEADER
I20260620 15:21:30.483352 21609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:30.483290 21619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe6862f41e4f30b31ebb3a3703e6dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } }
I20260620 15:21:30.486186 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12fe6862f41e4f30b31ebb3a3703e6dd. Latest consensus state: current_term: 1 leader_uuid: "12fe6862f41e4f30b31ebb3a3703e6dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe6862f41e4f30b31ebb3a3703e6dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } } }
I20260620 15:21:30.486203 21620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12fe6862f41e4f30b31ebb3a3703e6dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fe6862f41e4f30b31ebb3a3703e6dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39337 } } }
I20260620 15:21:30.486667 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:30.486667 21620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:30.488408 21626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:30.494619 21626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:30.504252 21626 catalog_manager.cc:1347] Generated new cluster ID: 376d516bde4a47cca23a08cd285b4a1a
I20260620 15:21:30.504354 21626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:30.524838 21626 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:30.526544 21626 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:30.535408 21626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12fe6862f41e4f30b31ebb3a3703e6dd: Generated new TSK 0
I20260620 15:21:30.536288 21626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:30.551844 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21542
W20260620 15:21:30.553150 21604 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.12:48513
W20260620 15:21:30.553884 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21542: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:30.554026 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 21542
W20260620 15:21:30.571394 21527 connection.cc:535] client connection to 127.0.0.1:39337 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39337): Connection reset by peer (error 104)
2026-06-20T15:21:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1856 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:30 dist-test-slave-7s64 krb5kdc[21640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:30 dist-test-slave-7s64 krb5kdc[21640](info): set up 2 sockets
Jun 20 15:21:30 dist-test-slave-7s64 krb5kdc[21640](info): commencing operation
krb5kdc: starting...
W20260620 15:21:31.644867 389 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:31 dist-test-slave-7s64 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:31.883407 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38813
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:32.199877 21653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:32.200129 21653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:32.200285 21653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:32.207378 21653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:32.207477 21653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:32.207566 21653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:32.207643 21653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:32.207720 21653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:32.217247 21653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:32.218413 21653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:32.219633 21653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:32.231105 21659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:32.231321 21661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:32.231761 21658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:32.231910 21653 server_base.cc:1034] running on GCE node
I20260620 15:21:32.232728 21653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:32.234066 21653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:32.235291 21653 hybrid_clock.cc:648] HybridClock initialized: now 1781968892235248 us; error 41 us; skew 500 ppm
I20260620 15:21:32.235620 21653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:32 dist-test-slave-7s64 krb5kdc[21640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:32.243332 21653 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:32.244740 21653 webserver.cc:460] Webserver started at http://127.0.0.1:35963/ using document root <none> and password file <none>
I20260620 15:21:32.245357 21653 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:32.245450 21653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:32.245672 21653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:32.248412 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "63e8b359b0b94b5dab6162642d0b0534"
format_stamp: "Formatted at 2026-06-20 15:21:32 on dist-test-slave-7s64"
server_key: "38a379133e8b08b21a40226e5460d21d"
server_key_iv: "fcf7788397573bb6a3d8351d5b70d19c"
server_key_version: "encryptionkey@0"
I20260620 15:21:32.249171 21653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "63e8b359b0b94b5dab6162642d0b0534"
format_stamp: "Formatted at 2026-06-20 15:21:32 on dist-test-slave-7s64"
server_key: "38a379133e8b08b21a40226e5460d21d"
server_key_iv: "fcf7788397573bb6a3d8351d5b70d19c"
server_key_version: "encryptionkey@0"
I20260620 15:21:32.253826 21653 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 15:21:32.256991 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:32.258217 21653 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:21:32.258373 21653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "63e8b359b0b94b5dab6162642d0b0534"
format_stamp: "Formatted at 2026-06-20 15:21:32 on dist-test-slave-7s64"
server_key: "38a379133e8b08b21a40226e5460d21d"
server_key_iv: "fcf7788397573bb6a3d8351d5b70d19c"
server_key_version: "encryptionkey@0"
I20260620 15:21:32.258560 21653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:32.263871 21653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:32.267670 21653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:32.267953 21653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:32.286185 21653 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38813
I20260620 15:21:32.286263 21719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38813 every 8 connection(s)
I20260620 15:21:32.287607 21653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:32.291308 21720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:32.294816 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 21653
I20260620 15:21:32.294991 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:32.295428 389 external_mini_cluster.cc:1428] Setting key 1289533914a12298306a08447e4af837
I20260620 15:21:32.302784 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:32.297207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:58465 (local address 127.0.0.1:38813)
0620 15:21:32.297925 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:32.297937 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:32.297974 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:21:32.299595 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:32.299615 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:32.299925 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:32.300305 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:32.301345 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:32.301351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:32.301362 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:32.301375 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:32.301425 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:32.301442 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:32.301664 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:32.301790 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 15:21:32.301953 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":314,"threads_started":1}
I20260620 15:21:32.303407 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534: Bootstrap starting.
I20260620 15:21:32.306331 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:32.306789 21720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:32.307736 21720 log.cc:824] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:32.310390 21720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534: No bootstrap required, opened a new log
I20260620 15:21:32.313321 21723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:32.310769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:38341 (local address 127.0.0.1:38813)
0620 15:21:32.310977 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:32.310985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:32.311431 (+ 446us) server_negotiation.cc:407] Connection header received
0620 15:21:32.311576 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:32.311580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:32.311621 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:32.311701 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:32.312648 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:32.312658 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:32.312662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:32.312678 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:32.312714 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:32.312732 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:32.312823 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:32.312954 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 15:21:32.312997 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:21:32.318495 21720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e8b359b0b94b5dab6162642d0b0534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260620 15:21:32.318862 21720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:32.318969 21720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e8b359b0b94b5dab6162642d0b0534, State: Initialized, Role: FOLLOWER
I20260620 15:21:32.319564 21720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e8b359b0b94b5dab6162642d0b0534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260620 15:21:32.319795 21720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:32.319943 21720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:32.320111 21720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:32.322425 21720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e8b359b0b94b5dab6162642d0b0534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260620 15:21:32.322968 21720 leader_election.cc:302] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e8b359b0b94b5dab6162642d0b0534; no voters:
I20260620 15:21:32.323593 21720 leader_election.cc:288] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:32.323678 21730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:32.324745 21730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [term 1 LEADER]: Becoming Leader. State: Replica: 63e8b359b0b94b5dab6162642d0b0534, State: Running, Role: LEADER
I20260620 15:21:32.325268 21730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e8b359b0b94b5dab6162642d0b0534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } }
I20260620 15:21:32.325563 21720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:32.328056 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63e8b359b0b94b5dab6162642d0b0534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e8b359b0b94b5dab6162642d0b0534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } } }
I20260620 15:21:32.328446 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:32.328652 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e8b359b0b94b5dab6162642d0b0534. Latest consensus state: current_term: 1 leader_uuid: "63e8b359b0b94b5dab6162642d0b0534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e8b359b0b94b5dab6162642d0b0534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38813 } } }
I20260620 15:21:32.329048 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:32.331666 21738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:32.335971 21738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:32.344695 21738 catalog_manager.cc:1347] Generated new cluster ID: 0820d60075a944b5856f571b3d8d370e
I20260620 15:21:32.344839 21738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:32.367359 21738 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:32.367758 21738 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:32.375295 21738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63e8b359b0b94b5dab6162642d0b0534: Generated new TSK 0
I20260620 15:21:32.376289 21738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:32.392431 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21653
W20260620 15:21:32.393651 21715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.12:58465
W20260620 15:21:32.394326 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21653: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:32.394484 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 21653
2026-06-20T15:21:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1839 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:32 dist-test-slave-7s64 krb5kdc[21751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:32 dist-test-slave-7s64 krb5kdc[21751](info): set up 2 sockets
Jun 20 15:21:32 dist-test-slave-7s64 krb5kdc[21751](info): commencing operation
krb5kdc: starting...
W20260620 15:21:33.500892 389 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:33 dist-test-slave-7s64 krb5kdc[21751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:33.733397 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:34.052538 21764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:34.052832 21764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:34.052923 21764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:34.059742 21764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:34.059834 21764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:34.059880 21764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:34.059937 21764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:34.059980 21764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:34.069258 21764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:34.070405 21764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:34.071580 21764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:34.083283 21769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:34.083190 21770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:34.083240 21772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:34.084144 21764 server_base.cc:1034] running on GCE node
I20260620 15:21:34.084931 21764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:34.086215 21764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:34.087410 21764 hybrid_clock.cc:648] HybridClock initialized: now 1781968894087378 us; error 29 us; skew 500 ppm
I20260620 15:21:34.087749 21764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:34 dist-test-slave-7s64 krb5kdc[21751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:34.090713 21764 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:34.092139 21764 webserver.cc:460] Webserver started at http://127.0.0.1:37201/ using document root <none> and password file <none>
I20260620 15:21:34.092865 21764 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:34.092967 21764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:34.093305 21764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:34.096160 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "a7f80aaf54084c1c89b964e3c83a228b"
format_stamp: "Formatted at 2026-06-20 15:21:34 on dist-test-slave-7s64"
server_key: "b8a610f15f291cb9b8a342c03e01055b"
server_key_iv: "ce86fee3e2b069f6fc6425cd1d06dda9"
server_key_version: "encryptionkey@0"
I20260620 15:21:34.096928 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "a7f80aaf54084c1c89b964e3c83a228b"
format_stamp: "Formatted at 2026-06-20 15:21:34 on dist-test-slave-7s64"
server_key: "b8a610f15f291cb9b8a342c03e01055b"
server_key_iv: "ce86fee3e2b069f6fc6425cd1d06dda9"
server_key_version: "encryptionkey@0"
I20260620 15:21:34.101888 21764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:21:34.105172 21779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:34.106545 21764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:21:34.106766 21764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "a7f80aaf54084c1c89b964e3c83a228b"
format_stamp: "Formatted at 2026-06-20 15:21:34 on dist-test-slave-7s64"
server_key: "b8a610f15f291cb9b8a342c03e01055b"
server_key_iv: "ce86fee3e2b069f6fc6425cd1d06dda9"
server_key_version: "encryptionkey@0"
I20260620 15:21:34.106982 21764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:34.140736 21764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:34.145270 21764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:34.145594 21764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:34.164799 21764 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35885
I20260620 15:21:34.164839 21830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35885 every 8 connection(s)
I20260620 15:21:34.166134 21764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:34.169945 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:34.174592 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 21764
I20260620 15:21:34.174790 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:34.175343 389 external_mini_cluster.cc:1428] Setting key 928c3adb75033693928968ea142b2f71
I20260620 15:21:34.181210 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b: Bootstrap starting.
I20260620 15:21:34.184754 21831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:34.185963 21831 log.cc:824] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:34.189405 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b: No bootstrap required, opened a new log
Jun 20 15:21:34 dist-test-slave-7s64 krb5kdc[21751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:21:34.200178 21831 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f80aaf54084c1c89b964e3c83a228b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260620 15:21:34.200688 21831 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:34.200815 21831 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f80aaf54084c1c89b964e3c83a228b, State: Initialized, Role: FOLLOWER
I20260620 15:21:34.201004 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:34.177297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:45825 (local address 127.0.0.1:35885)
0620 15:21:34.178135 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:34.178150 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:34.178197 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:21:34.180030 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:34.180051 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:34.180459 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:34.180857 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:34.182578 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:34.183967 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:34.185854 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:34.186908 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:34.187139 (+ 231us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:34.191904 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:34.191914 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:34.191936 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:34.191995 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:34.196328 (+ 4333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:34.197203 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:34.197213 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:34.197221 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:34.197431 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:34.197868 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:34.197877 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:34.197881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:34.198607 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:34.198932 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:34.199323 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 15:21:34.199636 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":390,"threads_started":1}
I20260620 15:21:34.201459 21831 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f80aaf54084c1c89b964e3c83a228b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260620 15:21:34.201722 21831 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:34.201865 21831 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:34.202049 21831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:34.204460 21831 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f80aaf54084c1c89b964e3c83a228b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260620 15:21:34.205035 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:34.205049 21831 leader_election.cc:302] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7f80aaf54084c1c89b964e3c83a228b; no voters:
I20260620 15:21:34.205631 21831 leader_election.cc:288] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:34.205756 21836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:34.206382 21836 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [term 1 LEADER]: Becoming Leader. State: Replica: a7f80aaf54084c1c89b964e3c83a228b, State: Running, Role: LEADER
I20260620 15:21:34.206947 21836 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f80aaf54084c1c89b964e3c83a228b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } }
I20260620 15:21:34.208643 21836 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f80aaf54084c1c89b964e3c83a228b. Latest consensus state: current_term: 1 leader_uuid: "a7f80aaf54084c1c89b964e3c83a228b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f80aaf54084c1c89b964e3c83a228b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } } }
I20260620 15:21:34.208983 21836 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:34.209471 21838 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7f80aaf54084c1c89b964e3c83a228b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7f80aaf54084c1c89b964e3c83a228b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35885 } } }
I20260620 15:21:34.209826 21838 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:34.221853 21831 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:34.225481 21841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:34.230808 21841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:34.241365 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:34.221081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:38157 (local address 127.0.0.1:35885)
0620 15:21:34.221283 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:34.221289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:34.222663 (+ 1374us) server_negotiation.cc:407] Connection header received
0620 15:21:34.222841 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:34.222849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:34.222956 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:34.223074 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:34.225434 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:34.226636 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:34.228565 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:34.229303 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:34.229413 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:34.232837 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:34.232844 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:34.232848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:34.232890 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:34.236632 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:34.237648 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:34.237652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:34.237655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:34.237805 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:34.238432 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:34.238439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:34.238446 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:34.239047 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:34.239316 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:34.240639 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0620 15:21:34.240859 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:21:34.244830 21841 catalog_manager.cc:1347] Generated new cluster ID: 76c89afe90114e72b12f21f14ae23047
I20260620 15:21:34.244930 21841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:34.266726 21841 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:34.267915 21841 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:34.278177 21841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f80aaf54084c1c89b964e3c83a228b: Generated new TSK 0
I20260620 15:21:34.279081 21841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:34.293020 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21764
I20260620 15:21:34.294647 21826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:34.386307 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 21764
W20260620 15:21:34.403291 21749 connection.cc:535] client connection to 127.0.0.1:35885 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35885): Connection reset by peer (error 104)
2026-06-20T15:21:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:34 dist-test-slave-7s64 krb5kdc[21863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:34 dist-test-slave-7s64 krb5kdc[21863](info): set up 2 sockets
Jun 20 15:21:34 dist-test-slave-7s64 krb5kdc[21863](info): commencing operation
krb5kdc: starting...
W20260620 15:21:35.488834 389 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:35 dist-test-slave-7s64 krb5kdc[21863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:35.715523 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:36.029148 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:36.029464 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:36.029645 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:36.036580 21876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:36.036722 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:36.036834 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:36.036929 21876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:36.036994 21876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:36.046294 21876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46855
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:36.047490 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:36.048787 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:36.060179 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:36.060240 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:36.060307 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:36.061347 21876 server_base.cc:1034] running on GCE node
I20260620 15:21:36.061998 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:36.063184 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:36.064385 21876 hybrid_clock.cc:648] HybridClock initialized: now 1781968896064347 us; error 40 us; skew 500 ppm
I20260620 15:21:36.064731 21876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:36 dist-test-slave-7s64 krb5kdc[21863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:36.067857 21876 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:36.069257 21876 webserver.cc:460] Webserver started at http://127.0.0.1:43223/ using document root <none> and password file <none>
I20260620 15:21:36.069893 21876 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:36.070015 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:36.070323 21876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:36.073256 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "c5fd52dc2f264a02989a07ea5004ce98"
format_stamp: "Formatted at 2026-06-20 15:21:36 on dist-test-slave-7s64"
server_key: "212e62025af0a7db145b908b0a8c2381"
server_key_iv: "f2bf7343ffd89eb75a3d13972ab39620"
server_key_version: "encryptionkey@0"
I20260620 15:21:36.074000 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "c5fd52dc2f264a02989a07ea5004ce98"
format_stamp: "Formatted at 2026-06-20 15:21:36 on dist-test-slave-7s64"
server_key: "212e62025af0a7db145b908b0a8c2381"
server_key_iv: "f2bf7343ffd89eb75a3d13972ab39620"
server_key_version: "encryptionkey@0"
I20260620 15:21:36.078820 21876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:21:36.082082 21891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:36.083323 21876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:21:36.083488 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "c5fd52dc2f264a02989a07ea5004ce98"
format_stamp: "Formatted at 2026-06-20 15:21:36 on dist-test-slave-7s64"
server_key: "212e62025af0a7db145b908b0a8c2381"
server_key_iv: "f2bf7343ffd89eb75a3d13972ab39620"
server_key_version: "encryptionkey@0"
I20260620 15:21:36.083668 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:36.096789 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:36.100899 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:36.101162 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:36.119388 21876 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46855
I20260620 15:21:36.119421 21942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46855 every 8 connection(s)
I20260620 15:21:36.120774 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:36.124212 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:36.124665 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 21876
I20260620 15:21:36.124848 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:36.125280 389 external_mini_cluster.cc:1428] Setting key 0b04482870da8df13e71baa120a609ab
I20260620 15:21:36.137198 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98: Bootstrap starting.
I20260620 15:21:36.138536 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:36.126916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:42149 (local address 127.0.0.1:46855)
0620 15:21:36.127657 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:36.127667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:36.127722 (+ 55us) server_negotiation.cc:407] Connection header received
0620 15:21:36.129923 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:36.129942 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:36.130307 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:36.130612 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:36.131163 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:36.132351 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:36.134283 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:36.134946 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:36.135065 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:36.136344 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:36.136351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:36.136362 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:36.136929 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:36.136997 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:36.137020 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:36.137131 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:36.137500 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 15:21:36.137656 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":359,"threads_started":1}
I20260620 15:21:36.140169 21943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:36.141422 21943 log.cc:824] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:36.142087 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:36.143905 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98: No bootstrap required, opened a new log
I20260620 15:21:36.152866 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:36.146356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:48619 (local address 127.0.0.1:46855)
0620 15:21:36.146568 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:36.146573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:36.146902 (+ 329us) server_negotiation.cc:407] Connection header received
0620 15:21:36.147037 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:36.147041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:36.147093 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:36.147175 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:36.147805 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:36.148578 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:36.150418 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:36.150843 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:36.150930 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:36.152112 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:36.152116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:36.152119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:36.152136 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:36.152168 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:36.152180 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:36.152264 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:36.152461 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 15:21:36.152548 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:21:36.153410 21943 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fd52dc2f264a02989a07ea5004ce98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46855 } }
I20260620 15:21:36.153779 21943 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:36.153880 21943 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5fd52dc2f264a02989a07ea5004ce98, State: Initialized, Role: FOLLOWER
I20260620 15:21:36.154397 21943 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fd52dc2f264a02989a07ea5004ce98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46855 } }
I20260620 15:21:36.154639 21943 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:36.154791 21943 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:36.154951 21943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:36.157104 21943 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fd52dc2f264a02989a07ea5004ce98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46855 } }
I20260620 15:21:36.157564 21943 leader_election.cc:302] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5fd52dc2f264a02989a07ea5004ce98; no voters:
I20260620 15:21:36.158068 21943 leader_election.cc:288] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:36.158206 21953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:36.159782 21953 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [term 1 LEADER]: Becoming Leader. State: Replica: c5fd52dc2f264a02989a07ea5004ce98, State: Running, Role: LEADER
I20260620 15:21:36.160260 21943 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:36.160285 21953 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fd52dc2f264a02989a07ea5004ce98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46855 } }
I20260620 15:21:36.162906 21954 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5fd52dc2f264a02989a07ea5004ce98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fd52dc2f264a02989a07ea5004ce98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46855 } } }
I20260620 15:21:36.163237 21954 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:36.163692 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5fd52dc2f264a02989a07ea5004ce98. Latest consensus state: current_term: 1 leader_uuid: "c5fd52dc2f264a02989a07ea5004ce98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5fd52dc2f264a02989a07ea5004ce98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46855 } } }
I20260620 15:21:36.164001 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:36.165115 21960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:36.170199 21960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:36.179124 21960 catalog_manager.cc:1347] Generated new cluster ID: 132a5befe93c4b09864fbac969c26092
I20260620 15:21:36.179267 21960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:36.199853 21960 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:36.201225 21960 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:36.209141 21960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5fd52dc2f264a02989a07ea5004ce98: Generated new TSK 0
I20260620 15:21:36.209910 21960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:36.219532 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21876
W20260620 15:21:36.220682 21938 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.12:42149
W20260620 15:21:36.221433 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21876: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:36.221593 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 21876
W20260620 15:21:36.237986 21861 connection.cc:535] client connection to 127.0.0.1:46855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46855): Connection reset by peer (error 104)
2026-06-20T15:21:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1834 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:36 dist-test-slave-7s64 krb5kdc[21974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:36 dist-test-slave-7s64 krb5kdc[21974](info): set up 2 sockets
Jun 20 15:21:36 dist-test-slave-7s64 krb5kdc[21974](info): commencing operation
krb5kdc: starting...
W20260620 15:21:37.312840 389 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:37 dist-test-slave-7s64 krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:21:37.532963 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:37.846329 21987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:37.846586 21987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:37.846745 21987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:37.853727 21987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:37.853822 21987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:37.853904 21987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:37.854001 21987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:37.854079 21987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:37.863448 21987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:37.864519 21987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:37.865718 21987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:37.877133 21992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:37.877135 21993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:37.877079 21995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:37.877751 21987 server_base.cc:1034] running on GCE node
I20260620 15:21:37.878418 21987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:37.879652 21987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:37.880900 21987 hybrid_clock.cc:648] HybridClock initialized: now 1781968897880860 us; error 32 us; skew 500 ppm
I20260620 15:21:37.881261 21987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:37 dist-test-slave-7s64 krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:37.887264 21987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:21:37.888734 21987 webserver.cc:460] Webserver started at http://127.0.0.1:45351/ using document root <none> and password file <none>
I20260620 15:21:37.889377 21987 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:37.889521 21987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:37.889802 21987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:37.892782 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "a2266475b386405bbc4317cab5765702"
format_stamp: "Formatted at 2026-06-20 15:21:37 on dist-test-slave-7s64"
server_key: "ae41f3ea9437f114c09752a947b24fa3"
server_key_iv: "d21ad9f517b592dd375322f4c3113ce4"
server_key_version: "encryptionkey@0"
I20260620 15:21:37.893534 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "a2266475b386405bbc4317cab5765702"
format_stamp: "Formatted at 2026-06-20 15:21:37 on dist-test-slave-7s64"
server_key: "ae41f3ea9437f114c09752a947b24fa3"
server_key_iv: "d21ad9f517b592dd375322f4c3113ce4"
server_key_version: "encryptionkey@0"
I20260620 15:21:37.898321 21987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:21:37.901679 22002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:37.902920 21987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:21:37.903095 21987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "a2266475b386405bbc4317cab5765702"
format_stamp: "Formatted at 2026-06-20 15:21:37 on dist-test-slave-7s64"
server_key: "ae41f3ea9437f114c09752a947b24fa3"
server_key_iv: "d21ad9f517b592dd375322f4c3113ce4"
server_key_version: "encryptionkey@0"
I20260620 15:21:37.903277 21987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:37.908815 21987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:37.912896 21987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:37.913189 21987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:37.930895 21987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42669
I20260620 15:21:37.930935 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42669 every 8 connection(s)
I20260620 15:21:37.932132 21987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:37.935559 22054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:37.940932 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 21987
I20260620 15:21:37.941213 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:37.941665 389 external_mini_cluster.cc:1428] Setting key 846bd9c0be1ddb3eeabd78836d986589
I20260620 15:21:37.945987 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702: Bootstrap starting.
I20260620 15:21:37.949064 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:37.943263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:35177 (local address 127.0.0.1:42669)
0620 15:21:37.944080 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:37.944094 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:37.944135 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:21:37.945784 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:37.945804 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:37.946154 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:37.946509 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:37.947577 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:37.947583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:37.947595 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:37.947610 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:37.947662 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:37.947684 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:37.947835 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:37.947942 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 15:21:37.948117 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":367,"threads_started":1}
I20260620 15:21:37.949554 22054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:37.950532 22054 log.cc:824] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:37.952710 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:37.953459 22054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702: No bootstrap required, opened a new log
I20260620 15:21:37.960021 22054 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2266475b386405bbc4317cab5765702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } }
I20260620 15:21:37.960354 22054 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:37.960482 22054 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2266475b386405bbc4317cab5765702, State: Initialized, Role: FOLLOWER
I20260620 15:21:37.961143 22054 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2266475b386405bbc4317cab5765702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } }
I20260620 15:21:37.961436 22054 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:37.961615 22054 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:37.961800 22054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:37.963759 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:37.961377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:35625 (local address 127.0.0.1:42669)
0620 15:21:37.961563 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:37.961569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:37.961944 (+ 375us) server_negotiation.cc:407] Connection header received
0620 15:21:37.962131 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:37.962135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:37.962176 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:37.962289 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:37.963146 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:37.963152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:37.963157 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:37.963174 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:37.963219 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:37.963234 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:37.963363 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:37.963440 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 15:21:37.963497 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:21:37.964083 22054 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2266475b386405bbc4317cab5765702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } }
I20260620 15:21:37.964620 22054 leader_election.cc:302] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2266475b386405bbc4317cab5765702; no voters:
I20260620 15:21:37.965260 22054 leader_election.cc:288] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:37.965349 22064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:37.966040 22064 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [term 1 LEADER]: Becoming Leader. State: Replica: a2266475b386405bbc4317cab5765702, State: Running, Role: LEADER
I20260620 15:21:37.966621 22064 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2266475b386405bbc4317cab5765702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } }
I20260620 15:21:37.967372 22054 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:37.969375 22065 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2266475b386405bbc4317cab5765702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2266475b386405bbc4317cab5765702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } } }
I20260620 15:21:37.969795 22065 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:37.971189 22066 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2266475b386405bbc4317cab5765702. Latest consensus state: current_term: 1 leader_uuid: "a2266475b386405bbc4317cab5765702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2266475b386405bbc4317cab5765702" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42669 } } }
I20260620 15:21:37.971544 22066 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:37.972126 22073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:37.977002 22073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:37.985193 22073 catalog_manager.cc:1347] Generated new cluster ID: b51e1a713d934523880294590805cb1b
I20260620 15:21:37.985309 22073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:37.998298 22073 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:37.998788 22073 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:38.011044 22073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2266475b386405bbc4317cab5765702: Generated new TSK 0
I20260620 15:21:38.011782 22073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:38.024545 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21987
W20260620 15:21:38.025586 22049 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.12:35177
W20260620 15:21:38.026212 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 21987: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:38.026365 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 21987
2026-06-20T15:21:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1805 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:38 dist-test-slave-7s64 krb5kdc[22085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:38 dist-test-slave-7s64 krb5kdc[22085](info): set up 2 sockets
Jun 20 15:21:38 dist-test-slave-7s64 krb5kdc[22085](info): commencing operation
krb5kdc: starting...
W20260620 15:21:39.116988 389 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:39 dist-test-slave-7s64 krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:21:39.343411 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43589
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:39.672896 22098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:39.673149 22098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:39.673237 22098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:39.680136 22098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:39.680227 22098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:39.680272 22098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:39.680323 22098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:39.680367 22098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:39.689715 22098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:39.690793 22098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:39.691994 22098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:39.703423 22103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:39.703584 22104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:39.703735 22106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:39.704473 22098 server_base.cc:1034] running on GCE node
I20260620 15:21:39.705227 22098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:39.706491 22098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:39.707731 22098 hybrid_clock.cc:648] HybridClock initialized: now 1781968899707688 us; error 56 us; skew 500 ppm
I20260620 15:21:39.708077 22098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:39 dist-test-slave-7s64 krb5kdc[22085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:39.711102 22098 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:21:39.712441 22098 webserver.cc:460] Webserver started at http://127.0.97.126:40761/ using document root <none> and password file <none>
I20260620 15:21:39.713120 22098 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:39.713217 22098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:39.713454 22098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:39.716116 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "3d4db056c61b4094b8b16c61c446c199"
format_stamp: "Formatted at 2026-06-20 15:21:39 on dist-test-slave-7s64"
server_key: "6d2918aacd64df843ef6c4a278027ef6"
server_key_iv: "bb2429b4117e3f586f5c7208e394e82e"
server_key_version: "encryptionkey@0"
I20260620 15:21:39.716864 22098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "3d4db056c61b4094b8b16c61c446c199"
format_stamp: "Formatted at 2026-06-20 15:21:39 on dist-test-slave-7s64"
server_key: "6d2918aacd64df843ef6c4a278027ef6"
server_key_iv: "bb2429b4117e3f586f5c7208e394e82e"
server_key_version: "encryptionkey@0"
I20260620 15:21:39.721596 22098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 15:21:39.724843 22113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:39.726082 22098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:21:39.726263 22098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "3d4db056c61b4094b8b16c61c446c199"
format_stamp: "Formatted at 2026-06-20 15:21:39 on dist-test-slave-7s64"
server_key: "6d2918aacd64df843ef6c4a278027ef6"
server_key_iv: "bb2429b4117e3f586f5c7208e394e82e"
server_key_version: "encryptionkey@0"
I20260620 15:21:39.726436 22098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:39.744786 22098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:39.748812 22098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:39.749082 22098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:39.766422 22098 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43589
I20260620 15:21:39.766492 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43589 every 8 connection(s)
I20260620 15:21:39.767633 22098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:39.770996 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:39.772919 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 22098
I20260620 15:21:39.773110 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:39.773554 389 external_mini_cluster.cc:1428] Setting key 47033280e74ef5ae14dcee88522854dc
I20260620 15:21:39.784468 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199: Bootstrap starting.
I20260620 15:21:39.787524 22165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199: Neither blocks nor log segments found. Creating new log.
Jun 20 15:21:39 dist-test-slave-7s64 krb5kdc[22085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:39.788771 22165 log.cc:824] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:39.791292 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199: No bootstrap required, opened a new log
I20260620 15:21:39.796726 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:39.775197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35885 (local address 127.0.97.126:43589)
0620 15:21:39.775941 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:39.775957 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:39.776016 (+ 59us) server_negotiation.cc:407] Connection header received
0620 15:21:39.777755 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:39.777790 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:39.778263 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:39.778635 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:39.779856 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:39.781911 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:39.783203 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:39.784178 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:39.784354 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:39.788940 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:39.788950 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:39.788964 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:39.789004 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:39.792295 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:39.793227 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:39.793234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:39.793240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:39.793410 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:39.793903 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:39.793907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:39.793909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:39.794546 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:39.794833 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:39.795280 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 15:21:39.795617 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":301,"threads_started":1}
I20260620 15:21:39.799796 22165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4db056c61b4094b8b16c61c446c199" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43589 } }
I20260620 15:21:39.800159 22165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:39.800279 22165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4db056c61b4094b8b16c61c446c199, State: Initialized, Role: FOLLOWER
I20260620 15:21:39.800554 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:39.800946 22165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4db056c61b4094b8b16c61c446c199" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43589 } }
I20260620 15:21:39.801206 22165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:39.801327 22165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:39.801486 22165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:39.804637 22165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4db056c61b4094b8b16c61c446c199" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43589 } }
I20260620 15:21:39.805191 22165 leader_election.cc:302] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d4db056c61b4094b8b16c61c446c199; no voters:
I20260620 15:21:39.805881 22165 leader_election.cc:288] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:39.806035 22174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:39.807806 22165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:39.813653 22174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [term 1 LEADER]: Becoming Leader. State: Replica: 3d4db056c61b4094b8b16c61c446c199, State: Running, Role: LEADER
I20260620 15:21:39.814294 22174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4db056c61b4094b8b16c61c446c199" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43589 } }
I20260620 15:21:39.817351 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d4db056c61b4094b8b16c61c446c199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4db056c61b4094b8b16c61c446c199" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43589 } } }
I20260620 15:21:39.817737 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:39.818454 22177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d4db056c61b4094b8b16c61c446c199. Latest consensus state: current_term: 1 leader_uuid: "3d4db056c61b4094b8b16c61c446c199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4db056c61b4094b8b16c61c446c199" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43589 } } }
I20260620 15:21:39.818773 22177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:39.819497 22189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:39.824409 22189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:39.827605 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:39.805522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56395 (local address 127.0.97.126:43589)
0620 15:21:39.805740 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:39.805749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:39.812059 (+ 6310us) server_negotiation.cc:407] Connection header received
0620 15:21:39.812276 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:39.812285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:39.812369 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:39.812464 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:39.814573 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:39.815708 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:39.817376 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:39.817985 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:39.818142 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:39.821817 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:39.821822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:39.821825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:39.821853 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:39.824783 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:39.825507 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:39.825510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:39.825512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:39.825620 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:39.826096 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:39.826099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:39.826101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:39.826392 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:39.826671 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:39.827064 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 15:21:39.827230 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:21:39.833278 22189 catalog_manager.cc:1347] Generated new cluster ID: 1fc2ceb4044f46d08ae5484917bae638
I20260620 15:21:39.833402 22189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:39.848125 22189 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:39.849323 22189 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:39.857264 22189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d4db056c61b4094b8b16c61c446c199: Generated new TSK 0
I20260620 15:21:39.858173 22189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:39.868974 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22098
I20260620 15:21:39.870059 22160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:39.960198 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 22098
W20260620 15:21:39.977095 22083 connection.cc:535] client connection to 127.0.97.126:43589 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.126:43589): Connection reset by peer (error 104)
2026-06-20T15:21:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:40 dist-test-slave-7s64 krb5kdc[22197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:40 dist-test-slave-7s64 krb5kdc[22197](info): set up 2 sockets
Jun 20 15:21:40 dist-test-slave-7s64 krb5kdc[22197](info): commencing operation
krb5kdc: starting...
W20260620 15:21:41.072824 389 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:41 dist-test-slave-7s64 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:21:41.293677 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44325
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:41.607827 22210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:41.608167 22210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:41.608327 22210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:41.615139 22210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:41.615268 22210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:41.615361 22210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:41.615463 22210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:41.615538 22210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:41.625067 22210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:41.626189 22210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:41.627414 22210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:41.638646 22215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:41.638952 22218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:41.638940 22216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:41.639770 22210 server_base.cc:1034] running on GCE node
I20260620 15:21:41.640640 22210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:41.641911 22210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:41.643134 22210 hybrid_clock.cc:648] HybridClock initialized: now 1781968901643098 us; error 60 us; skew 500 ppm
I20260620 15:21:41.643463 22210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:41 dist-test-slave-7s64 krb5kdc[22197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:41.646427 22210 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:21:41.647866 22210 webserver.cc:460] Webserver started at http://127.0.97.126:46033/ using document root <none> and password file <none>
I20260620 15:21:41.648561 22210 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:41.648708 22210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:41.648991 22210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:41.651746 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "0a56ae0542e643068b7e04d6e745107e"
format_stamp: "Formatted at 2026-06-20 15:21:41 on dist-test-slave-7s64"
server_key: "18aebd7ab3a4c9860efb187de67dac78"
server_key_iv: "5bd7504ea603e33413884c654ceb5db5"
server_key_version: "encryptionkey@0"
I20260620 15:21:41.652479 22210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "0a56ae0542e643068b7e04d6e745107e"
format_stamp: "Formatted at 2026-06-20 15:21:41 on dist-test-slave-7s64"
server_key: "18aebd7ab3a4c9860efb187de67dac78"
server_key_iv: "5bd7504ea603e33413884c654ceb5db5"
server_key_version: "encryptionkey@0"
I20260620 15:21:41.657374 22210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:21:41.660542 22225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:41.661803 22210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:21:41.662010 22210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "0a56ae0542e643068b7e04d6e745107e"
format_stamp: "Formatted at 2026-06-20 15:21:41 on dist-test-slave-7s64"
server_key: "18aebd7ab3a4c9860efb187de67dac78"
server_key_iv: "5bd7504ea603e33413884c654ceb5db5"
server_key_version: "encryptionkey@0"
I20260620 15:21:41.662210 22210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:41.687703 22210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:41.691380 22210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:41.691637 22210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:41.709605 22210 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44325
I20260620 15:21:41.709661 22276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44325 every 8 connection(s)
I20260620 15:21:41.710834 22210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:41.714303 22277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:41.714862 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 22210
I20260620 15:21:41.715030 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:41.715611 389 external_mini_cluster.cc:1428] Setting key 32849750998ee3ac24d13257cc578652
I20260620 15:21:41.726956 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e: Bootstrap starting.
I20260620 15:21:41.729116 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:41.717299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40751 (local address 127.0.97.126:44325)
0620 15:21:41.718070 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:41.718081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:41.718122 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:21:41.720845 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:41.720868 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:41.721230 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:41.721739 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:41.722262 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:41.723463 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:41.724857 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:41.725522 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:41.725695 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:41.726918 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:41.726925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:41.726937 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:41.727486 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:41.727555 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:41.727578 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:41.727737 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:41.728010 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 15:21:41.728175 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":353,"threads_started":1}
I20260620 15:21:41.730084 22277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:41.731370 22277 log.cc:824] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:41.732872 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:41.733937 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e: No bootstrap required, opened a new log
I20260620 15:21:41.743556 22277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a56ae0542e643068b7e04d6e745107e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44325 } }
I20260620 15:21:41.743925 22277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:41.744065 22277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a56ae0542e643068b7e04d6e745107e, State: Initialized, Role: FOLLOWER
I20260620 15:21:41.744648 22277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a56ae0542e643068b7e04d6e745107e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44325 } }
I20260620 15:21:41.744903 22277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:41.745064 22277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:41.745280 22277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:41.746591 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:41.737003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43365 (local address 127.0.97.126:44325)
0620 15:21:41.737183 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:41.737187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:41.741007 (+ 3820us) server_negotiation.cc:407] Connection header received
0620 15:21:41.741208 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:41.741212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:41.741266 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:41.741367 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:41.741822 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:41.742604 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:41.743945 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:41.744614 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:41.744753 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:41.745871 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:41.745881 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:41.745890 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:41.745905 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:41.745951 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:41.745968 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:41.746100 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:41.746278 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 15:21:41.746332 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:21:41.747686 22277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a56ae0542e643068b7e04d6e745107e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44325 } }
I20260620 15:21:41.748215 22277 leader_election.cc:302] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a56ae0542e643068b7e04d6e745107e; no voters:
I20260620 15:21:41.748790 22277 leader_election.cc:288] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:41.748915 22287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:41.750121 22287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [term 1 LEADER]: Becoming Leader. State: Replica: 0a56ae0542e643068b7e04d6e745107e, State: Running, Role: LEADER
I20260620 15:21:41.750640 22287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a56ae0542e643068b7e04d6e745107e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44325 } }
I20260620 15:21:41.750856 22277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:41.753255 22289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a56ae0542e643068b7e04d6e745107e. Latest consensus state: current_term: 1 leader_uuid: "0a56ae0542e643068b7e04d6e745107e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a56ae0542e643068b7e04d6e745107e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44325 } } }
I20260620 15:21:41.753755 22289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:41.753712 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a56ae0542e643068b7e04d6e745107e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a56ae0542e643068b7e04d6e745107e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44325 } } }
I20260620 15:21:41.753990 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:41.754607 22295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:41.759909 22295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:41.768998 22295 catalog_manager.cc:1347] Generated new cluster ID: 9cd7394cc6a9429ab9b9a0cc317ec628
I20260620 15:21:41.769106 22295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:41.780830 22295 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:41.781970 22295 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:41.787817 22295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a56ae0542e643068b7e04d6e745107e: Generated new TSK 0
I20260620 15:21:41.788684 22295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:41.802253 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22210
W20260620 15:21:41.803278 22272 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40751
W20260620 15:21:41.803949 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22210: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:41.804085 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 22210
W20260620 15:21:41.820574 22195 connection.cc:535] client connection to 127.0.97.126:44325 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.126:44325): Connection reset by peer (error 104)
2026-06-20T15:21:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1842 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:41 dist-test-slave-7s64 krb5kdc[22308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:41 dist-test-slave-7s64 krb5kdc[22308](info): set up 2 sockets
Jun 20 15:21:41 dist-test-slave-7s64 krb5kdc[22308](info): commencing operation
krb5kdc: starting...
W20260620 15:21:42.892858 389 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:43 dist-test-slave-7s64 krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:21:43.118994 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36039
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:43.434352 22321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:43.434602 22321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:43.434688 22321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:43.441416 22321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:43.441519 22321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:43.441561 22321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:43.441612 22321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:43.441654 22321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:43.450683 22321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36039
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:43.451728 22321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:43.452911 22321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:43.464243 22326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:43.464201 22327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:43.464385 22329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:43.464820 22321 server_base.cc:1034] running on GCE node
I20260620 15:21:43.465682 22321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:43.466924 22321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:43.468145 22321 hybrid_clock.cc:648] HybridClock initialized: now 1781968903468114 us; error 47 us; skew 500 ppm
I20260620 15:21:43.468473 22321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:43 dist-test-slave-7s64 krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:43.476578 22321 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:21:43.477942 22321 webserver.cc:460] Webserver started at http://127.0.97.126:38893/ using document root <none> and password file <none>
I20260620 15:21:43.478585 22321 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:43.478724 22321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:43.479015 22321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:43.481868 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "0b376ca4a7664a12a0eab81cebfafc6f"
format_stamp: "Formatted at 2026-06-20 15:21:43 on dist-test-slave-7s64"
server_key: "ec7e6dda1a7c820d24faec66098c350c"
server_key_iv: "c5a0b44de800a0de976f41dc725bfeb5"
server_key_version: "encryptionkey@0"
I20260620 15:21:43.482601 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "0b376ca4a7664a12a0eab81cebfafc6f"
format_stamp: "Formatted at 2026-06-20 15:21:43 on dist-test-slave-7s64"
server_key: "ec7e6dda1a7c820d24faec66098c350c"
server_key_iv: "c5a0b44de800a0de976f41dc725bfeb5"
server_key_version: "encryptionkey@0"
I20260620 15:21:43.487270 22321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:21:43.490463 22336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:43.491652 22321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:21:43.491830 22321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "0b376ca4a7664a12a0eab81cebfafc6f"
format_stamp: "Formatted at 2026-06-20 15:21:43 on dist-test-slave-7s64"
server_key: "ec7e6dda1a7c820d24faec66098c350c"
server_key_iv: "c5a0b44de800a0de976f41dc725bfeb5"
server_key_version: "encryptionkey@0"
I20260620 15:21:43.492012 22321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:43.497385 22321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:43.501010 22321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:43.501282 22321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:43.518816 22321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36039
I20260620 15:21:43.518888 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36039 every 8 connection(s)
I20260620 15:21:43.520040 22321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:43.522114 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 22321
I20260620 15:21:43.522301 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:43.522890 389 external_mini_cluster.cc:1428] Setting key c65447f03056a8270ed0c64c23a61f26
I20260620 15:21:43.524433 22388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:43.529996 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:43.524626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38897 (local address 127.0.97.126:36039)
0620 15:21:43.525358 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:43.525369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:43.525404 (+ 35us) server_negotiation.cc:407] Connection header received
0620 15:21:43.526823 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:43.526841 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:43.527147 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:43.527443 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:43.528458 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:43.528464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:43.528474 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:43.528536 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:43.528618 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:43.528641 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:43.528804 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:43.528971 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 15:21:43.529138 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":347,"threads_started":1}
I20260620 15:21:43.533560 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:43.537267 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f: Bootstrap starting.
I20260620 15:21:43.540249 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:43.537680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32853 (local address 127.0.97.126:36039)
0620 15:21:43.537885 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:43.537895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:43.538405 (+ 510us) server_negotiation.cc:407] Connection header received
0620 15:21:43.538520 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:43.538525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:43.538580 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:43.538714 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:43.539566 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:43.539571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:43.539576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:43.539594 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:43.539636 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:43.539660 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:43.539790 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:43.539896 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 15:21:43.539951 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 15:21:43.540992 22388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:43.542073 22388 log.cc:824] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:43.544631 22388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f: No bootstrap required, opened a new log
I20260620 15:21:43.551978 22388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b376ca4a7664a12a0eab81cebfafc6f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36039 } }
I20260620 15:21:43.552328 22388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:43.552451 22388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b376ca4a7664a12a0eab81cebfafc6f, State: Initialized, Role: FOLLOWER
I20260620 15:21:43.553130 22388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b376ca4a7664a12a0eab81cebfafc6f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36039 } }
I20260620 15:21:43.553375 22388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:43.553537 22388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:43.553716 22388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:43.555848 22388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b376ca4a7664a12a0eab81cebfafc6f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36039 } }
I20260620 15:21:43.556351 22388 leader_election.cc:302] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b376ca4a7664a12a0eab81cebfafc6f; no voters:
I20260620 15:21:43.556979 22388 leader_election.cc:288] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:43.557106 22398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:43.557967 22398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [term 1 LEADER]: Becoming Leader. State: Replica: 0b376ca4a7664a12a0eab81cebfafc6f, State: Running, Role: LEADER
I20260620 15:21:43.558502 22398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b376ca4a7664a12a0eab81cebfafc6f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36039 } }
I20260620 15:21:43.559044 22388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:43.561170 22399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b376ca4a7664a12a0eab81cebfafc6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b376ca4a7664a12a0eab81cebfafc6f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36039 } } }
I20260620 15:21:43.561360 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b376ca4a7664a12a0eab81cebfafc6f. Latest consensus state: current_term: 1 leader_uuid: "0b376ca4a7664a12a0eab81cebfafc6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b376ca4a7664a12a0eab81cebfafc6f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36039 } } }
I20260620 15:21:43.561641 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:43.561641 22399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:43.564644 22405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:43.570137 22405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:43.578250 22405 catalog_manager.cc:1347] Generated new cluster ID: 46cfaaec8aa9427a9d714a26ecca24bf
I20260620 15:21:43.578359 22405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:43.595805 22405 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:43.596328 22405 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:43.613054 22405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b376ca4a7664a12a0eab81cebfafc6f: Generated new TSK 0
I20260620 15:21:43.613996 22405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:43.619683 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22321
W20260620 15:21:43.620855 22383 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38897
W20260620 15:21:43.621598 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22321: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:43.621753 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 22321
2026-06-20T15:21:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1817 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:43 dist-test-slave-7s64 krb5kdc[22419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:43 dist-test-slave-7s64 krb5kdc[22419](info): set up 2 sockets
Jun 20 15:21:43 dist-test-slave-7s64 krb5kdc[22419](info): commencing operation
krb5kdc: starting...
W20260620 15:21:44.701053 389 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:44 dist-test-slave-7s64 krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:21:44.925662 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42263
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:45.245076 22432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:45.245328 22432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:45.245416 22432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:45.252351 22432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:45.252441 22432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:45.252527 22432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:45.252611 22432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:45.252656 22432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:45.261914 22432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:45.263001 22432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:45.264176 22432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:45.275377 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:45.275509 22437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:45.275446 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:45.276182 22432 server_base.cc:1034] running on GCE node
I20260620 15:21:45.276988 22432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:45.278308 22432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:45.279533 22432 hybrid_clock.cc:648] HybridClock initialized: now 1781968905279497 us; error 48 us; skew 500 ppm
I20260620 15:21:45.279861 22432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:45 dist-test-slave-7s64 krb5kdc[22419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:45.282919 22432 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:21:45.284278 22432 webserver.cc:460] Webserver started at http://127.0.97.126:34823/ using document root <none> and password file <none>
I20260620 15:21:45.284951 22432 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:45.285086 22432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:45.285389 22432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:45.288154 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "a315f538a04e4b7c9dbf8e97641cf0d2"
format_stamp: "Formatted at 2026-06-20 15:21:45 on dist-test-slave-7s64"
server_key: "dab618df71be86b5dce2d3cce60b65e3"
server_key_iv: "95ac3d74d319cfd91931a48edef263bf"
server_key_version: "encryptionkey@0"
I20260620 15:21:45.288942 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "a315f538a04e4b7c9dbf8e97641cf0d2"
format_stamp: "Formatted at 2026-06-20 15:21:45 on dist-test-slave-7s64"
server_key: "dab618df71be86b5dce2d3cce60b65e3"
server_key_iv: "95ac3d74d319cfd91931a48edef263bf"
server_key_version: "encryptionkey@0"
I20260620 15:21:45.293637 22432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:21:45.296859 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:45.298013 22432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260620 15:21:45.298192 22432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "a315f538a04e4b7c9dbf8e97641cf0d2"
format_stamp: "Formatted at 2026-06-20 15:21:45 on dist-test-slave-7s64"
server_key: "dab618df71be86b5dce2d3cce60b65e3"
server_key_iv: "95ac3d74d319cfd91931a48edef263bf"
server_key_version: "encryptionkey@0"
I20260620 15:21:45.298395 22432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:45.312891 22432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:45.316977 22432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:45.317239 22432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:45.335287 22432 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42263
I20260620 15:21:45.335314 22498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42263 every 8 connection(s)
I20260620 15:21:45.336509 22432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:45.337415 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 22432
I20260620 15:21:45.337594 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:45.337980 389 external_mini_cluster.cc:1428] Setting key f09c32f55b94ac9ff6c8f9e6cc214fc9
I20260620 15:21:45.340598 22499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:21:45 dist-test-slave-7s64 krb5kdc[22419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:45.352344 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2: Bootstrap starting.
I20260620 15:21:45.356211 22499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:45.357455 22499 log.cc:824] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:45.360646 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2: No bootstrap required, opened a new log
I20260620 15:21:45.361527 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:45.339587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56909 (local address 127.0.97.126:42263)
0620 15:21:45.340362 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:45.340377 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:45.340436 (+ 59us) server_negotiation.cc:407] Connection header received
0620 15:21:45.342683 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:45.342708 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:45.343148 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:45.343573 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:45.344818 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:45.346791 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:45.348023 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:45.349007 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:45.349193 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:45.353005 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:45.353015 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:45.353044 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:45.353092 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:45.356949 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:45.357718 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:45.357724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:45.357730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:45.357989 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:45.358515 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:45.358518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:45.358521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:45.359150 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:45.359415 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:45.359835 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 15:21:45.360220 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":332,"threads_started":1}
I20260620 15:21:45.365176 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:45.368682 22499 raft_consensus.cc:348] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315f538a04e4b7c9dbf8e97641cf0d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42263 } }
I20260620 15:21:45.369050 22499 raft_consensus.cc:374] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:45.369158 22499 raft_consensus.cc:729] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a315f538a04e4b7c9dbf8e97641cf0d2, State: Initialized, Role: FOLLOWER
I20260620 15:21:45.369876 22499 consensus_queue.cc:260] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315f538a04e4b7c9dbf8e97641cf0d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42263 } }
I20260620 15:21:45.370148 22499 raft_consensus.cc:388] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:45.370313 22499 raft_consensus.cc:482] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:45.370512 22499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:45.372648 22499 raft_consensus.cc:504] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315f538a04e4b7c9dbf8e97641cf0d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42263 } }
I20260620 15:21:45.373095 22499 leader_election.cc:302] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a315f538a04e4b7c9dbf8e97641cf0d2; no voters:
I20260620 15:21:45.373733 22499 leader_election.cc:288] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:45.373834 22509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:45.376024 22499 sys_catalog.cc:564] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:45.381206 22509 raft_consensus.cc:686] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [term 1 LEADER]: Becoming Leader. State: Replica: a315f538a04e4b7c9dbf8e97641cf0d2, State: Running, Role: LEADER
I20260620 15:21:45.382009 22509 consensus_queue.cc:237] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315f538a04e4b7c9dbf8e97641cf0d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42263 } }
I20260620 15:21:45.382441 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:45.369540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46375 (local address 127.0.97.126:42263)
0620 15:21:45.369710 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:45.369723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:45.370083 (+ 360us) server_negotiation.cc:407] Connection header received
0620 15:21:45.370271 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:45.370277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:45.370356 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:45.370454 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:45.371761 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:45.372559 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:45.373921 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:45.374626 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:45.374742 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:45.377042 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:45.377048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:45.377052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:45.377088 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:45.379800 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:45.380405 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:45.380411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:45.380414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:45.380593 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:45.380970 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:45.380976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:45.380980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:45.381245 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:45.381554 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:45.381842 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 15:21:45.382047 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 15:21:45.385433 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a315f538a04e4b7c9dbf8e97641cf0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315f538a04e4b7c9dbf8e97641cf0d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42263 } } }
I20260620 15:21:45.385871 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:45.386471 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a315f538a04e4b7c9dbf8e97641cf0d2. Latest consensus state: current_term: 1 leader_uuid: "a315f538a04e4b7c9dbf8e97641cf0d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a315f538a04e4b7c9dbf8e97641cf0d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42263 } } }
I20260620 15:21:45.386780 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:45.388047 22517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:45.393543 22517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:45.401909 22517 catalog_manager.cc:1347] Generated new cluster ID: 3530f0a6e8db478dac15a4647f161f3d
I20260620 15:21:45.402027 22517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:45.424763 22517 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:45.426573 22517 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:45.436236 22517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a315f538a04e4b7c9dbf8e97641cf0d2: Generated new TSK 0
I20260620 15:21:45.437160 22517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:45.444031 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22432
I20260620 15:21:45.445063 22494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:45.534489 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 22432
2026-06-20T15:21:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1910 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:45 dist-test-slave-7s64 krb5kdc[22532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:45 dist-test-slave-7s64 krb5kdc[22532](info): set up 2 sockets
Jun 20 15:21:45 dist-test-slave-7s64 krb5kdc[22532](info): commencing operation
krb5kdc: starting...
W20260620 15:21:46.684860 389 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:46 dist-test-slave-7s64 krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:21:46.905006 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36391
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:47.224102 22545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:47.224359 22545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:47.224550 22545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:47.231518 22545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:47.231608 22545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:47.231741 22545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:47.231856 22545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:47.231936 22545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:47.241168 22545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:47.242311 22545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:47.243512 22545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:47.254837 22553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:47.254959 22551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:47.254916 22550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:47.255959 22545 server_base.cc:1034] running on GCE node
I20260620 15:21:47.256765 22545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:47.258004 22545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:47.259198 22545 hybrid_clock.cc:648] HybridClock initialized: now 1781968907259160 us; error 28 us; skew 500 ppm
I20260620 15:21:47.259475 22545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:47 dist-test-slave-7s64 krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:47.262579 22545 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:21:47.263990 22545 webserver.cc:460] Webserver started at http://127.0.97.126:43931/ using document root <none> and password file <none>
I20260620 15:21:47.264755 22545 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:47.264896 22545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:47.265223 22545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:47.268218 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "611065f6e03b467cad1f843a15e72015"
format_stamp: "Formatted at 2026-06-20 15:21:47 on dist-test-slave-7s64"
server_key: "bf57987b25486ff62c4b53d0100b4543"
server_key_iv: "4c7cadfcf775d2d590053672ee81f2d8"
server_key_version: "encryptionkey@0"
I20260620 15:21:47.269066 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "611065f6e03b467cad1f843a15e72015"
format_stamp: "Formatted at 2026-06-20 15:21:47 on dist-test-slave-7s64"
server_key: "bf57987b25486ff62c4b53d0100b4543"
server_key_iv: "4c7cadfcf775d2d590053672ee81f2d8"
server_key_version: "encryptionkey@0"
I20260620 15:21:47.274140 22545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:21:47.277550 22560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:47.278795 22545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:21:47.278980 22545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "611065f6e03b467cad1f843a15e72015"
format_stamp: "Formatted at 2026-06-20 15:21:47 on dist-test-slave-7s64"
server_key: "bf57987b25486ff62c4b53d0100b4543"
server_key_iv: "4c7cadfcf775d2d590053672ee81f2d8"
server_key_version: "encryptionkey@0"
I20260620 15:21:47.279168 22545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:47.295071 22545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:47.298789 22545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:47.299063 22545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:47.317152 22545 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36391
I20260620 15:21:47.317198 22611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36391 every 8 connection(s)
I20260620 15:21:47.318413 22545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:47.321861 22612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:47.325294 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 22545
I20260620 15:21:47.325454 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:47.325865 389 external_mini_cluster.cc:1428] Setting key 957db2510f6245dc066179fa3a216f69
I20260620 15:21:47.333552 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015: Bootstrap starting.
I20260620 15:21:47.337026 22612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:47.338102 22612 log.cc:824] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:47.338464 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:47.327516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44709 (local address 127.0.97.126:36391)
0620 15:21:47.328247 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:47.328257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:47.328295 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:21:47.329905 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:47.329925 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:47.330289 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:47.330579 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:47.331172 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:47.332433 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:47.334254 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:47.334876 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:47.335002 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:47.336237 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:47.336244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:47.336256 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:47.336806 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:47.336876 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:47.336899 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:47.336986 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:47.337253 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 15:21:47.337525 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":359,"threads_started":1}
I20260620 15:21:47.340749 22612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015: No bootstrap required, opened a new log
I20260620 15:21:47.341810 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:47.350246 22612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611065f6e03b467cad1f843a15e72015" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36391 } }
I20260620 15:21:47.350587 22612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:47.350694 22612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 611065f6e03b467cad1f843a15e72015, State: Initialized, Role: FOLLOWER
I20260620 15:21:47.351339 22612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611065f6e03b467cad1f843a15e72015" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36391 } }
I20260620 15:21:47.351625 22612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:47.351778 22612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:47.351958 22612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:47.353850 22615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:47.346783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51323 (local address 127.0.97.126:36391)
0620 15:21:47.346957 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:47.346962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:47.347477 (+ 515us) server_negotiation.cc:407] Connection header received
0620 15:21:47.347648 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:47.347653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:47.347719 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:47.347818 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:47.348313 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:47.349443 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:47.350953 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:47.351621 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:47.351767 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:47.353022 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:47.353028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:47.353034 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:47.353054 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:47.353100 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:47.353120 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:47.353233 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:47.353418 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 15:21:47.353511 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:21:47.355237 22612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611065f6e03b467cad1f843a15e72015" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36391 } }
I20260620 15:21:47.355700 22612 leader_election.cc:302] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 611065f6e03b467cad1f843a15e72015; no voters:
I20260620 15:21:47.356186 22612 leader_election.cc:288] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:47.356333 22622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:47.357041 22622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [term 1 LEADER]: Becoming Leader. State: Replica: 611065f6e03b467cad1f843a15e72015, State: Running, Role: LEADER
I20260620 15:21:47.357589 22622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611065f6e03b467cad1f843a15e72015" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36391 } }
I20260620 15:21:47.358224 22612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:47.361248 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "611065f6e03b467cad1f843a15e72015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611065f6e03b467cad1f843a15e72015" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36391 } } }
I20260620 15:21:47.361229 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 611065f6e03b467cad1f843a15e72015. Latest consensus state: current_term: 1 leader_uuid: "611065f6e03b467cad1f843a15e72015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611065f6e03b467cad1f843a15e72015" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36391 } } }
I20260620 15:21:47.361668 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:47.361683 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:47.363035 22631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:47.368016 22631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:47.376016 22631 catalog_manager.cc:1347] Generated new cluster ID: 42ace8aa8eef4c1f865893d6ecb4dabe
I20260620 15:21:47.376194 22631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:47.395530 22631 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:47.396818 22631 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:47.405179 22631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 611065f6e03b467cad1f843a15e72015: Generated new TSK 0
I20260620 15:21:47.405926 22631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:47.416255 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22545
W20260620 15:21:47.417512 22607 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44709
W20260620 15:21:47.418116 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22545: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:47.418252 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 22545
2026-06-20T15:21:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:47 dist-test-slave-7s64 krb5kdc[22643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:47 dist-test-slave-7s64 krb5kdc[22643](info): set up 2 sockets
Jun 20 15:21:47 dist-test-slave-7s64 krb5kdc[22643](info): commencing operation
krb5kdc: starting...
W20260620 15:21:48.508787 389 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:48 dist-test-slave-7s64 krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:21:48.733646 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42327
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:49.044854 22656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:49.045120 22656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:49.045296 22656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:49.052415 22656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:49.052531 22656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:49.052618 22656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:49.052732 22656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:49.052814 22656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:49.062157 22656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:49.063346 22656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:49.064596 22656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:49.075832 22664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:49.075937 22661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:49.075893 22662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:49.077029 22656 server_base.cc:1034] running on GCE node
I20260620 15:21:49.077816 22656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:49.079159 22656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:49.080451 22656 hybrid_clock.cc:648] HybridClock initialized: now 1781968909080458 us; error 100 us; skew 500 ppm
I20260620 15:21:49.080884 22656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:49 dist-test-slave-7s64 krb5kdc[22643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:49.087080 22656 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:21:49.088471 22656 webserver.cc:460] Webserver started at http://127.0.97.126:42457/ using document root <none> and password file <none>
I20260620 15:21:49.089149 22656 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:49.089246 22656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:49.089494 22656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:49.092250 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "4cbc5b17769349dabb9ae90023223d6a"
format_stamp: "Formatted at 2026-06-20 15:21:49 on dist-test-slave-7s64"
server_key: "d15e9d49826a027193a255cc145a5272"
server_key_iv: "810f3e4d5750bdb944f392a0a4d88ca9"
server_key_version: "encryptionkey@0"
I20260620 15:21:49.093014 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "4cbc5b17769349dabb9ae90023223d6a"
format_stamp: "Formatted at 2026-06-20 15:21:49 on dist-test-slave-7s64"
server_key: "d15e9d49826a027193a255cc145a5272"
server_key_iv: "810f3e4d5750bdb944f392a0a4d88ca9"
server_key_version: "encryptionkey@0"
I20260620 15:21:49.097726 22656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 15:21:49.100973 22671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:49.102174 22656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:21:49.102334 22656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "4cbc5b17769349dabb9ae90023223d6a"
format_stamp: "Formatted at 2026-06-20 15:21:49 on dist-test-slave-7s64"
server_key: "d15e9d49826a027193a255cc145a5272"
server_key_iv: "810f3e4d5750bdb944f392a0a4d88ca9"
server_key_version: "encryptionkey@0"
I20260620 15:21:49.102507 22656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:49.107877 22656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:49.111900 22656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:49.112179 22656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:49.129911 22656 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42327
I20260620 15:21:49.129987 22722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42327 every 8 connection(s)
I20260620 15:21:49.131261 22656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:49.133195 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 22656
I20260620 15:21:49.133381 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:49.133996 389 external_mini_cluster.cc:1428] Setting key fb74b763a840285bb9887fe63e707858
I20260620 15:21:49.135879 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:49.145710 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:49.136338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47177 (local address 127.0.97.126:42327)
0620 15:21:49.139310 (+ 2972us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:49.139326 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:49.139372 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:21:49.142133 (+ 2761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:49.142160 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:49.142581 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:49.143029 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:49.144298 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:49.144304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:49.144315 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:49.144326 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:49.144378 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:49.144397 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:49.144625 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:49.144754 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 15:21:49.144938 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2710,"thread_start_us":2368,"threads_started":1}
I20260620 15:21:49.149147 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:49.150238 22723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a: Bootstrap starting.
I20260620 15:21:49.153391 22723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:49.154496 22723 log.cc:824] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:49.155854 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:49.153279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57097 (local address 127.0.97.126:42327)
0620 15:21:49.153426 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:49.153430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:49.153909 (+ 479us) server_negotiation.cc:407] Connection header received
0620 15:21:49.154089 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:49.154094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:49.154147 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:49.154246 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:49.155220 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:49.155226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:49.155231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:21:49.155251 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:49.155304 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:21:49.155328 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:49.155456 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:49.155528 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 15:21:49.155581 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 15:21:49.157346 22723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a: No bootstrap required, opened a new log
I20260620 15:21:49.164458 22723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbc5b17769349dabb9ae90023223d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42327 } }
I20260620 15:21:49.164870 22723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:21:49.164979 22723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbc5b17769349dabb9ae90023223d6a, State: Initialized, Role: FOLLOWER
I20260620 15:21:49.165540 22723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbc5b17769349dabb9ae90023223d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42327 } }
I20260620 15:21:49.165834 22723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:21:49.165995 22723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:21:49.166178 22723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:49.168169 22723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbc5b17769349dabb9ae90023223d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42327 } }
I20260620 15:21:49.168635 22723 leader_election.cc:302] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cbc5b17769349dabb9ae90023223d6a; no voters:
I20260620 15:21:49.169113 22723 leader_election.cc:288] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:21:49.169257 22733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:49.169924 22733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [term 1 LEADER]: Becoming Leader. State: Replica: 4cbc5b17769349dabb9ae90023223d6a, State: Running, Role: LEADER
I20260620 15:21:49.170461 22733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbc5b17769349dabb9ae90023223d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42327 } }
I20260620 15:21:49.170993 22723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:49.172991 22735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cbc5b17769349dabb9ae90023223d6a. Latest consensus state: current_term: 1 leader_uuid: "4cbc5b17769349dabb9ae90023223d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbc5b17769349dabb9ae90023223d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42327 } } }
I20260620 15:21:49.173377 22735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:49.173645 22734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cbc5b17769349dabb9ae90023223d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cbc5b17769349dabb9ae90023223d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42327 } } }
I20260620 15:21:49.173980 22734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:49.177606 22740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:49.183012 22740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:49.191128 22740 catalog_manager.cc:1347] Generated new cluster ID: 408ce75bbbc440c9905a5ca0aef7c278
I20260620 15:21:49.191233 22740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:49.216004 22740 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:49.216379 22740 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:49.231379 22740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cbc5b17769349dabb9ae90023223d6a: Generated new TSK 0
I20260620 15:21:49.232067 22740 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:49.241578 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22656
W20260620 15:21:49.242712 22718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47177
W20260620 15:21:49.243366 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22656: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:21:49.243516 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 22656
2026-06-20T15:21:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1828 ms)
[----------] 18 tests from AuthTokenIssuingTest (33681 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:49 dist-test-slave-7s64 krb5kdc[22754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:49 dist-test-slave-7s64 krb5kdc[22754](info): set up 2 sockets
Jun 20 15:21:49 dist-test-slave-7s64 krb5kdc[22754](info): commencing operation
krb5kdc: starting...
W20260620 15:21:50.337061 389 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:50 dist-test-slave-7s64 krb5kdc[22754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:21:50.562089 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37069
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37069,127.0.97.125:36561,127.0.97.124:35447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:50.876737 22767 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:21:50.877075 22767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:50.877202 22767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:50.877338 22767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:50.884305 22767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:50.884439 22767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:50.884570 22767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:50.884657 22767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:50.884747 22767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:50.894143 22767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37069,127.0.97.125:36561,127.0.97.124:35447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:50.895347 22767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:50.896586 22767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:50.907773 22772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:50.907805 22773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:50.907747 22775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:50.908742 22767 server_base.cc:1034] running on GCE node
I20260620 15:21:50.909418 22767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:50.910709 22767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:50.911921 22767 hybrid_clock.cc:648] HybridClock initialized: now 1781968910911855 us; error 44 us; skew 500 ppm
I20260620 15:21:50.912225 22767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:50 dist-test-slave-7s64 krb5kdc[22754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:50.915272 22767 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:21:50.916680 22767 webserver.cc:460] Webserver started at http://127.0.97.126:40163/ using document root <none> and password file <none>
I20260620 15:21:50.917309 22767 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:50.917457 22767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:50.917773 22767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:50.920643 22767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "5827af20c6dc4d058337bc64e0bdd09f"
format_stamp: "Formatted at 2026-06-20 15:21:50 on dist-test-slave-7s64"
server_key: "58e849b2afe172dd2ee5f339b9b39dfd"
server_key_iv: "2316230c0e8855b73dbfaa78e0837ced"
server_key_version: "encryptionkey@0"
I20260620 15:21:50.921401 22767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "5827af20c6dc4d058337bc64e0bdd09f"
format_stamp: "Formatted at 2026-06-20 15:21:50 on dist-test-slave-7s64"
server_key: "58e849b2afe172dd2ee5f339b9b39dfd"
server_key_iv: "2316230c0e8855b73dbfaa78e0837ced"
server_key_version: "encryptionkey@0"
I20260620 15:21:50.925987 22767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:21:50.929096 22782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:50.930230 22767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 15:21:50.930411 22767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "5827af20c6dc4d058337bc64e0bdd09f"
format_stamp: "Formatted at 2026-06-20 15:21:50 on dist-test-slave-7s64"
server_key: "58e849b2afe172dd2ee5f339b9b39dfd"
server_key_iv: "2316230c0e8855b73dbfaa78e0837ced"
server_key_version: "encryptionkey@0"
I20260620 15:21:50.930619 22767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:50.949088 22767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:50.953073 22767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:50.953329 22767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:50.970695 22767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37069
I20260620 15:21:50.970718 22833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37069 every 8 connection(s)
I20260620 15:21:50.971855 22767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:50.975100 22834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:50.980842 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 22767
I20260620 15:21:50.981060 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:50.981674 389 external_mini_cluster.cc:1428] Setting key 72c2639885cb58f704cfd9139399b7d7
I20260620 15:21:50.981432 22834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:21:50 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:51.003508 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:50.984371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37069 (local address 127.0.0.1:37194)
0620 15:21:50.985086 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:50.985120 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:50.986983 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:50.987713 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:50.987720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:50.988343 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:50.988711 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:50.988730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:50.990147 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:50.990153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:50.991163 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:50.991173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:50.992026 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:50.992032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:50.992257 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:50.992878 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:50.992899 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:50.994905 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:50.999240 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:50.999249 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:50.999257 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:50.999621 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.000090 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.000093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.000096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.000242 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.001269 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:51.001275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:51.001452 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 15:21:51.001689 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 15:21:51.002254 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":134,"thread_start_us":293,"threads_started":1}
I20260620 15:21:51.003543 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:50.985538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37194 (local address 127.0.97.126:37069)
0620 15:21:50.986207 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:50.986214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:50.986259 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:21:50.987312 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:50.987317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:50.987512 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:50.987704 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:50.988911 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:50.990004 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:50.991304 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:50.991818 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:50.992010 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:50.995141 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:50.995153 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:50.995170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:50.995212 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:50.999093 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:50.999769 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:50.999780 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:50.999787 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:50.999974 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.000374 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.000382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.000401 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.001131 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:51.001347 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:51.001809 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 15:21:51.002137 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":460,"thread_start_us":300,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.125@KRBTEST.COM; defaulting to no policy
I20260620 15:21:51.006464 22834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:21:51.007663 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.007225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51490)
0620 15:21:51.007373 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.007472 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.125@KRBTEST.COM" created.
W20260620 15:21:51.008167 22783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
I20260620 15:21:51.009909 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.009451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51494)
0620 15:21:51.009639 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.009733 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 15:21:51.010128 22834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:36561: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab.
I20260620 15:21:51.035607 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:36561
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37069,127.0.97.125:36561,127.0.97.124:35447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:21:51.059451 22834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } attempt: 1
I20260620 15:21:51.060686 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.060139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51496)
0620 15:21:51.060379 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.060528 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 15:21:51.061669 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.061314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51510)
0620 15:21:51.061504 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.061564 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 15:21:51.061848 22834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:36561: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
I20260620 15:21:51.130085 22834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } attempt: 2
I20260620 15:21:51.131188 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.130755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51524)
0620 15:21:51.130953 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.131051 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 15:21:51.132125 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.131790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51538)
0620 15:21:51.131933 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.132013 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:21:51.132309 22834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:36561: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
I20260620 15:21:51.223567 22834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } attempt: 3
I20260620 15:21:51.224783 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.224303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51546)
0620 15:21:51.224537 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.224636 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 15:21:51.225772 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.225424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51548)
0620 15:21:51.225569 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.225638 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:21:51.225932 22834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:36561: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
W20260620 15:21:51.353803 22841 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:21:51.354147 22841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:51.354272 22841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:51.354405 22841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:51.361403 22841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:51.361515 22841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:51.361617 22841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:51.361714 22841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:51.361800 22841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:51.369187 22834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } attempt: 4
I20260620 15:21:51.370473 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.369886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51562)
0620 15:21:51.370215 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.370318 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260620 15:21:51.371558 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.371163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51570)
0620 15:21:51.371308 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.371407 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:21:51.371806 22834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:36561: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36561: connect: Connection refused (error 111)
I20260620 15:21:51.371873 22841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37069,127.0.97.125:36561,127.0.97.124:35447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:36561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:51.373045 22841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:51.374298 22841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:51.385484 22846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:51.385660 22847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:51.386019 22849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:51.387049 22841 server_base.cc:1034] running on GCE node
I20260620 15:21:51.387759 22841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:51.389072 22841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:51.390316 22841 hybrid_clock.cc:648] HybridClock initialized: now 1781968911390258 us; error 37 us; skew 500 ppm
I20260620 15:21:51.390618 22841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:51 dist-test-slave-7s64 krb5kdc[22754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:51.393641 22841 init.cc:377] Logged in from keytab as kudu/127.0.97.125@KRBTEST.COM (short username kudu)
I20260620 15:21:51.395040 22841 webserver.cc:460] Webserver started at http://127.0.97.125:37023/ using document root <none> and password file <none>
I20260620 15:21:51.395677 22841 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:51.395823 22841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:51.396126 22841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:51.398900 22841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/data/instance:
uuid: "5491d3eb740649e5b874b4fd99bed5a3"
format_stamp: "Formatted at 2026-06-20 15:21:51 on dist-test-slave-7s64"
server_key: "5032e93c760d55624e293192da95d6c5"
server_key_iv: "4309e7383a478b05222ea9325fc635a1"
server_key_version: "encryptionkey@0"
I20260620 15:21:51.399652 22841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/wal/instance:
uuid: "5491d3eb740649e5b874b4fd99bed5a3"
format_stamp: "Formatted at 2026-06-20 15:21:51 on dist-test-slave-7s64"
server_key: "5032e93c760d55624e293192da95d6c5"
server_key_iv: "4309e7383a478b05222ea9325fc635a1"
server_key_version: "encryptionkey@0"
I20260620 15:21:51.404351 22841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260620 15:21:51.407553 22856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:51.408826 22841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:21:51.408991 22841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/wal
uuid: "5491d3eb740649e5b874b4fd99bed5a3"
format_stamp: "Formatted at 2026-06-20 15:21:51 on dist-test-slave-7s64"
server_key: "5032e93c760d55624e293192da95d6c5"
server_key_iv: "4309e7383a478b05222ea9325fc635a1"
server_key_version: "encryptionkey@0"
I20260620 15:21:51.409191 22841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:51.447013 22841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:51.450878 22841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:51.451153 22841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:51.469138 22841 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:36561
I20260620 15:21:51.469161 22907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:36561 every 8 connection(s)
I20260620 15:21:51.470393 22841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
I20260620 15:21:51.473896 22908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:51.476783 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 22841
I20260620 15:21:51.476965 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-1/wal/instance
I20260620 15:21:51.477494 389 external_mini_cluster.cc:1428] Setting key 7a18c3165c277f4864031bb8f0bffcef
I20260620 15:21:51.482452 22908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:21:51 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.124@KRBTEST.COM; defaulting to no policy
I20260620 15:21:51.504703 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.485593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.0.97.126:37069)
0620 15:21:51.485855 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:51.485864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:51.486964 (+ 1100us) server_negotiation.cc:407] Connection header received
0620 15:21:51.489253 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:51.489258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:51.489326 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:51.489408 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:51.491311 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.491988 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.494364 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.494801 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.494892 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.498856 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:51.498860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:51.498864 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:51.498901 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:51.501668 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.502401 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.502411 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.502413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.502532 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.502935 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.502940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.502943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.503336 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:51.503476 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:51.504148 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 15:21:51.504370 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.124@KRBTEST.COM" created.
I20260620 15:21:51.505656 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.486020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37069 (local address 127.0.0.1:37206)
0620 15:21:51.486794 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.486842 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:51.488950 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:51.489647 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:51.489660 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:51.490387 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:51.491144 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.491168 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.492145 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.492154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:51.494216 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.494233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.495012 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.495021 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:51.495326 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.496009 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:51.496042 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:51.498610 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:51.501840 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.501848 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.501861 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.502275 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.502644 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.502649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.502653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.502821 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.503445 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:51.503453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:51.503770 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 15:21:51.504016 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 15:21:51.504475 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":349,"threads_started":1}
I20260620 15:21:51.507316 22908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:21:51 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260620 15:21:51.523597 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.508118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.0.97.125:36561)
0620 15:21:51.508692 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:51.508697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:51.508720 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:21:51.508760 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:51.508765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:51.508916 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:51.509021 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:51.509953 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.510958 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.512931 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.513924 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.514065 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.517060 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:51.517074 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:51.517092 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:51.517138 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:51.520917 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.521552 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.521557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.521559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.521676 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.522099 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.522102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.522104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.522428 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:51.522561 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:51.523018 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 15:21:51.523321 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":440,"thread_start_us":278,"threads_started":1}
I20260620 15:21:51.523686 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.508043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51580)
0620 15:21:51.508233 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.508253 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:51.508354 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:51.509095 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:51.509103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:51.509589 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:51.509791 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.509804 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.511102 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.511111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:51.512736 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.512753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.514068 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.514072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:51.514205 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.514770 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:51.514790 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:51.516854 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:51.521037 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.521041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.521043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.521409 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.521788 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.521795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.521802 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.521991 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.522554 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:51.522562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:51.522729 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 15:21:51.522887 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 15:21:51.523204 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:21:51.526265 22908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:21:51.527220 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.526834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36874)
0620 15:21:51.526982 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.527103 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:21:51.527539 22857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
I20260620 15:21:51.529026 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.528742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36876)
0620 15:21:51.528869 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.528941 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:21:51.529330 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35447: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab.
I20260620 15:21:51.537083 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:35447
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37069,127.0.97.125:36561,127.0.97.124:35447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:21:51.578640 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } attempt: 1
I20260620 15:21:51.580030 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.579422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36884)
0620 15:21:51.579678 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.579839 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 15:21:51.581032 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.580742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36898)
0620 15:21:51.580873 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.580941 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:21:51.581235 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35447: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
I20260620 15:21:51.649444 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } attempt: 2
I20260620 15:21:51.650549 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.650093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36910)
0620 15:21:51.650298 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.650395 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:21:51.651551 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.651227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36920)
0620 15:21:51.651369 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.651443 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:21:51.651758 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35447: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
I20260620 15:21:51.671083 22834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } attempt: 5
Jun 20 15:21:51 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260620 15:21:51.683950 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.671828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.0.97.125:36561)
0620 15:21:51.672032 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:51.672038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:51.672125 (+ 87us) server_negotiation.cc:407] Connection header received
0620 15:21:51.672338 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:51.672344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:51.672417 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:51.672555 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:51.673308 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.674034 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.675191 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.675601 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.675738 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.678760 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:51.678765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:51.678768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:51.678798 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:51.681522 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.682137 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.682144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.682147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.682272 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.682622 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.682628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.682631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.682897 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:51.683045 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:51.683432 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 15:21:51.683638 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 15:21:51.683950 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.671794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51588)
0620 15:21:51.672009 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.672031 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:51.672203 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:51.672577 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:51.672581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:51.672993 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:51.673161 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.673169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.674167 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.674180 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:51.675035 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.675046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.675765 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.675771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:51.675948 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.676609 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:51.676626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:51.678570 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:51.681657 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.681662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.681667 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.682017 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.682381 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.682386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.682390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.682517 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.683013 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:51.683018 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:51.683171 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 15:21:51.683333 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 15:21:51.683602 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 15:21:51.685261 22834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:21:51.686246 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.685908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36926)
0620 15:21:51.686052 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.686119 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:21:51.687110 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.686815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36928)
0620 15:21:51.686946 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.687024 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:21:51.687296 22834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35447: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
I20260620 15:21:51.738515 22834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } attempt: 1
I20260620 15:21:51.739835 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.739363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36936)
0620 15:21:51.739565 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.739649 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 15:21:51.740844 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.740469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36948)
0620 15:21:51.740652 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.740741 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 15:21:51.741042 22834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35447: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
I20260620 15:21:51.742980 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } attempt: 3
I20260620 15:21:51.743917 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.743502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36962)
0620 15:21:51.743696 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.743779 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 15:21:51.744930 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.744543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36974)
0620 15:21:51.744744 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.744832 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260620 15:21:51.745154 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35447: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
I20260620 15:21:51.809242 22834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } attempt: 2
I20260620 15:21:51.810355 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.809868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36980)
0620 15:21:51.810111 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.810197 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 15:21:51.811280 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.810966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36982)
0620 15:21:51.811111 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.811186 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:21:51.811486 22834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35447: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
W20260620 15:21:51.854595 22915 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:21:51.854923 22915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:51.855044 22915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:51.855180 22915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:51.862152 22915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:51.862262 22915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:51.862388 22915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:51.862470 22915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:51.862555 22915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:51.871754 22915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37069,127.0.97.125:36561,127.0.97.124:35447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:35447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:51.872934 22915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:51.874181 22915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:51.885402 22923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:51.885483 22920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:51.885437 22921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:51.886227 22915 server_base.cc:1034] running on GCE node
I20260620 15:21:51.886979 22915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:51.888198 22915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:51.888373 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } attempt: 4
I20260620 15:21:51.889420 22915 hybrid_clock.cc:648] HybridClock initialized: now 1781968911889329 us; error 58 us; skew 500 ppm
I20260620 15:21:51.889585 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.889061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36986)
0620 15:21:51.889257 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.889369 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 15:21:51.889780 22915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:51.890566 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.890265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36992)
0620 15:21:51.890404 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.890471 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 15:21:51.890935 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35447: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Jun 20 15:21:51 dist-test-slave-7s64 krb5kdc[22754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:51.893054 22915 init.cc:377] Logged in from keytab as kudu/127.0.97.124@KRBTEST.COM (short username kudu)
I20260620 15:21:51.894374 22915 webserver.cc:460] Webserver started at http://127.0.97.124:34613/ using document root <none> and password file <none>
I20260620 15:21:51.894997 22915 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:51.895076 22915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:51.895330 22915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:51.898029 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/data/instance:
uuid: "03547dce9c95449586d00c8cb1d42e80"
format_stamp: "Formatted at 2026-06-20 15:21:51 on dist-test-slave-7s64"
server_key: "8e70e38e4e3a67fc1ba9a450efa27768"
server_key_iv: "9a7d14c04900156c7254dc83bba43019"
server_key_version: "encryptionkey@0"
I20260620 15:21:51.898747 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/wal/instance:
uuid: "03547dce9c95449586d00c8cb1d42e80"
format_stamp: "Formatted at 2026-06-20 15:21:51 on dist-test-slave-7s64"
server_key: "8e70e38e4e3a67fc1ba9a450efa27768"
server_key_iv: "9a7d14c04900156c7254dc83bba43019"
server_key_version: "encryptionkey@0"
I20260620 15:21:51.903375 22915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:21:51.906406 22930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:51.907604 22915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 15:21:51.907768 22915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/wal
uuid: "03547dce9c95449586d00c8cb1d42e80"
format_stamp: "Formatted at 2026-06-20 15:21:51 on dist-test-slave-7s64"
server_key: "8e70e38e4e3a67fc1ba9a450efa27768"
server_key_iv: "9a7d14c04900156c7254dc83bba43019"
server_key_version: "encryptionkey@0"
I20260620 15:21:51.907971 22915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:51.917695 22834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } attempt: 3
I20260620 15:21:51.918624 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.918219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:36994)
0620 15:21:51.918392 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.918475 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 15:21:51.919523 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.919195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:37006)
0620 15:21:51.919374 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.919436 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:21:51.919724 22834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:35447: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35447: connect: Connection refused (error 111)
I20260620 15:21:51.926944 22915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:51.930455 22915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:51.930701 22915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:51.947937 22915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:35447
I20260620 15:21:51.947950 22981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:35447 every 8 connection(s)
I20260620 15:21:51.949149 22915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
I20260620 15:21:51.952571 22982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:51.958487 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 22915
I20260620 15:21:51.958654 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781968566423581-389-0/minicluster-data/master-2/wal/instance
I20260620 15:21:51.959052 389 external_mini_cluster.cc:1428] Setting key a45ac9a464104dd631838e7ac5885d42
I20260620 15:21:51.958531 22982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:21:51 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
Jun 20 15:21:51 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260620 15:21:51.979609 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.961401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.0.97.126:37069)
0620 15:21:51.961600 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:51.961608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:51.962306 (+ 698us) server_negotiation.cc:407] Connection header received
0620 15:21:51.963926 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:51.963930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:51.964005 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:51.964085 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:51.965435 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.966174 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.967893 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.968619 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.968710 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.972818 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:51.972825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:51.972830 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:51.972869 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:51.976390 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.977091 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.977098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.977104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.977278 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.977791 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.977795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.977798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.978144 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:51.978305 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:51.979050 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 15:21:51.979266 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:21:51.980376 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.961312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37069 (local address 127.0.0.1:37222)
0620 15:21:51.962146 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.962188 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:51.963676 (+ 1488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:51.964171 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:51.964177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:51.964940 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:51.965262 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.965276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.966332 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.966337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:51.967746 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.967757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.968829 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.968841 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:51.969075 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.969847 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:51.969879 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:51.972628 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:51.976541 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.976553 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.976563 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.976967 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.977417 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.977422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.977426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.977636 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.978283 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:51.978290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:51.978683 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 15:21:51.978915 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 15:21:51.979224 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":443,"threads_started":1}
I20260620 15:21:51.981870 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.961069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.0.97.124:35447)
0620 15:21:51.961965 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:51.961978 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:51.962020 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:21:51.963676 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:51.963701 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:51.964004 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:51.964192 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:51.965578 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.967040 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.968566 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.969250 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.969397 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.974558 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:51.974566 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:51.974576 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:51.974637 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:51.978720 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.979498 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.979505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.979522 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.979664 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.980093 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.980097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.980099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.980454 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:51.980808 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:51.981354 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 15:21:51.981577 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":429,"threads_started":1}
I20260620 15:21:51.982123 22982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:21:51 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
Jun 20 15:21:51 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:51.995776 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.982852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36561 (local address 127.0.0.1:51602)
0620 15:21:51.983123 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.983136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:51.983257 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:51.983555 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:51.983559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:51.984012 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:51.984214 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.984225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.985651 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.985657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:51.986768 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.986779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.987456 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.987461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:51.987683 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.988192 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:51.988208 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:51.990017 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:51.993303 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.993307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.993309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.993774 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.994121 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:51.994128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:51.994132 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:51.994291 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:51.994777 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:51.994785 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:51.994978 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 15:21:51.995142 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 15:21:51.995424 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195}
I20260620 15:21:51.995781 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.982842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.0.97.125:36561)
0620 15:21:51.983023 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:51.983029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:51.983211 (+ 182us) server_negotiation.cc:407] Connection header received
0620 15:21:51.983369 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:51.983375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:51.983441 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:51.983538 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:51.984381 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.985447 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.986904 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.987321 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.987452 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.990197 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:51.990210 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:51.990213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:51.990245 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:51.993186 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.993890 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.993893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.993895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.994013 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.994401 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.994405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.994407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.994634 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:51.994759 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:51.995243 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 15:21:51.995394 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:21:51.997213 22982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:21:51.999683 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.984553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.0.97.126:37069)
0620 15:21:51.984712 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:51.984716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:51.984729 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:21:51.984765 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:51.984768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:51.984813 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:51.984888 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:51.986271 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.986932 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.988102 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:51.988594 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:51.988724 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:51.993553 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:51.993559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:51.993565 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:51.993605 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:51.996805 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.997612 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.997617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.997620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.997775 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:51.998219 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:51.998225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:51.998228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:51.998573 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:51.998846 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:51.999138 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 15:21:51.999350 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 15:21:51 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Jun 20 15:21:52 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260620 15:21:52.010264 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.997844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:37024)
0620 15:21:51.998030 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:51.998050 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:51.998139 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:51.998467 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:51.998472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:51.998939 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:51.999107 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:51.999117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.000279 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.000282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:52.001439 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:52.001450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.002120 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.002125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:52.002367 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:52.002906 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:52.002921 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:52.004689 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:52.007396 (+ 2707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:52.007402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:52.007407 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:52.007742 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:52.008666 (+ 924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:52.008673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:52.008679 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:52.008805 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:52.009308 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:52.009313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:52.009453 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 15:21:52.009596 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 15:21:52.009872 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":23}
I20260620 15:21:52.010293 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:51.998061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.0.97.124:35447)
0620 15:21:51.998219 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:51.998223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:51.998238 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:21:51.998277 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:51.998280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:51.998331 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:51.998412 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:51.999269 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.000144 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.001568 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.001981 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.002113 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:52.004871 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:52.004879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:52.004883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:52.004915 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:52.007273 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.008359 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.008366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.008372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.008547 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.008926 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.008931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.008935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.009196 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:52.009343 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:52.009678 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 15:21:52.010014 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":48}
I20260620 15:21:52.014771 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:52.001246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51616 (local address 127.0.97.125:36561)
0620 15:21:52.001402 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.001407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.001422 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:21:52.001520 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.001526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.001586 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.001691 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.002766 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.003418 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.004699 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.005105 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.005234 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:52.009171 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:52.009177 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:52.009181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:52.009208 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:52.012122 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.012885 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.012892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.012895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.013025 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.013417 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.013421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.013426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.013699 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:52.013987 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:52.014253 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 15:21:52.014470 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:21:52.015648 22982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80: Bootstrap starting.
I20260620 15:21:52.019385 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:52.019480 22982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:52.020596 22982 log.cc:824] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:52.023380 22982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80: No bootstrap required, opened a new log
I20260620 15:21:52.022207 22809 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5827af20c6dc4d058337bc64e0bdd09f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37238
I20260620 15:21:52.030225 22982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } }
I20260620 15:21:52.030550 22982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03547dce9c95449586d00c8cb1d42e80, State: Initialized, Role: FOLLOWER
I20260620 15:21:52.030970 22982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } }
I20260620 15:21:52.031841 22988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } } }
I20260620 15:21:52.032194 22988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:52.032900 22982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:21:52.041911 22999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:21:52.042030 22999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:21:52.039875 22915 master_runner.cc:186] Error getting master registration for 127.0.97.126:37069: OK, instance_id { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" instance_seqno: 1781968910965539 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:21:52.050812 22915 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:21:52.096998 22834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } attempt: 4
Jun 20 15:21:52 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260620 15:21:52.114008 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:52.097668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:37030)
0620 15:21:52.097969 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:52.097988 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:52.098112 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:52.098499 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:52.098505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:52.099033 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:52.099244 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:52.099254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.100757 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.100762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:52.102232 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:52.102248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.103249 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.103256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:52.103463 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:52.104464 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:52.104512 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:52.107272 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:52.111407 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:52.111427 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:52.111434 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:52.111940 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:52.112322 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:52.112326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:52.112330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:52.112525 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:52.113211 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:52.113214 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:52.113334 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 15:21:52.113444 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 15:21:52.113746 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 15:21:52.114172 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:52.097693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.0.97.124:35447)
0620 15:21:52.097947 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.097954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.098051 (+ 97us) server_negotiation.cc:407] Connection header received
0620 15:21:52.098247 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.098253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.098339 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.098460 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.099399 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.100615 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.102390 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.103100 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.103233 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:52.107455 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:52.107462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:52.107467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:52.107504 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:52.111265 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.112067 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.112073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.112076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.112209 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.112621 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.112626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.112630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.113093 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:52.113259 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:52.113582 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 15:21:52.113804 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:21:52.119990 22834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f: Bootstrap starting.
I20260620 15:21:52.124028 22834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:52.125324 22834 log.cc:824] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:52.128546 22834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f: No bootstrap required, opened a new log
I20260620 15:21:52.136065 22834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } }
I20260620 15:21:52.136453 22834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5827af20c6dc4d058337bc64e0bdd09f, State: Initialized, Role: FOLLOWER
I20260620 15:21:52.136984 22834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } }
I20260620 15:21:52.137954 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } } }
I20260620 15:21:52.138355 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:52.139163 22834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:52.139250 22809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 15:21:52.139446 22809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:52.142104 22809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } }
I20260620 15:21:52.143294 22809 leader_election.cc:288] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [CANDIDATE]: Term 1 election: Requested vote from peers 5491d3eb740649e5b874b4fd99bed5a3 (127.0.97.125:36561), 03547dce9c95449586d00c8cb1d42e80 (127.0.97.124:35447)
I20260620 15:21:52.144407 22957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5827af20c6dc4d058337bc64e0bdd09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "03547dce9c95449586d00c8cb1d42e80"
I20260620 15:21:52.144901 22957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:52.148929 22957 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5827af20c6dc4d058337bc64e0bdd09f in term 1.
I20260620 15:21:52.149712 22783 leader_election.cc:302] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03547dce9c95449586d00c8cb1d42e80, 5827af20c6dc4d058337bc64e0bdd09f; no voters:
I20260620 15:21:52.150194 23002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:52.161434 23002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [term 1 LEADER]: Becoming Leader. State: Replica: 5827af20c6dc4d058337bc64e0bdd09f, State: Running, Role: LEADER
I20260620 15:21:52.162132 23002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } }
I20260620 15:21:52.170674 23009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5827af20c6dc4d058337bc64e0bdd09f. Latest consensus state: current_term: 1 leader_uuid: "5827af20c6dc4d058337bc64e0bdd09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } } }
I20260620 15:21:52.171227 23009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:52.181408 23016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260620 15:21:52.182531 22936 tablet.cc:2369] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:21:52.190141 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } attempt: 5
I20260620 15:21:52.201407 23016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 15:21:52.206319 22788 tablet.cc:2369] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:21:52.208742 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:52.173792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.0.97.126:37069)
0620 15:21:52.173961 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.173966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.176550 (+ 2584us) server_negotiation.cc:407] Connection header received
0620 15:21:52.177061 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.177066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.177149 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.177239 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.185000 (+ 7761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.186374 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.188398 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.189222 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.189325 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:52.193466 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:52.193487 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:52.193495 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:52.193551 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:52.197766 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.206081 (+ 8315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.206086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.206090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.206309 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.206892 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.206900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.206904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.207273 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:52.207511 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:52.208074 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 15:21:52.208327 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:21:52.212342 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:52.177183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.0.97.124:35447)
0620 15:21:52.177361 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.177368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.187429 (+ 10061us) server_negotiation.cc:407] Connection header received
0620 15:21:52.187782 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.187787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.187863 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.187929 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.192088 (+ 4159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.193364 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.196845 (+ 3481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.197503 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.197602 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:52.202357 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:52.202366 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:52.202373 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:52.202415 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:52.206005 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.208169 (+ 2164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.208174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.208177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.208321 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.209016 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.209022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.209025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.209336 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:52.209557 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:52.211668 (+ 2111us) server_negotiation.cc:299] Negotiation successful
0620 15:21:52.211917 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:21:52.213231 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:52.175765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.0.97.125:36561)
0620 15:21:52.175981 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.175989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.181169 (+ 5180us) server_negotiation.cc:407] Connection header received
0620 15:21:52.181489 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.181498 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.181606 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.181694 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.192626 (+ 10932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.194075 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.200943 (+ 6868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.201798 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.201909 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:52.205567 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:52.205575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:52.205583 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:52.205635 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:52.209833 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.210676 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.210681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.210684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.210865 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.211308 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.211315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.211320 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.211690 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:52.211915 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:52.212683 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 15:21:52.212903 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:21:52.215528 22957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [term 1 FOLLOWER]: Refusing update from remote peer 5827af20c6dc4d058337bc64e0bdd09f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:52.217722 23009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Jun 20 15:21:52 dist-test-slave-7s64 krb5kdc[22754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260620 15:21:52.235267 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:52.190674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35447 (local address 127.0.0.1:37056)
0620 15:21:52.191068 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:52.191089 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:52.191186 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:52.210694 (+ 19508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:52.210698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:52.211173 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:52.211423 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:52.211435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.213358 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.213366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:52.214731 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:52.214744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.215915 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.215922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:52.216154 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:52.216873 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:52.216902 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:52.220526 (+ 3624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:52.232455 (+ 11929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:52.232460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:52.232464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:52.233010 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:52.233522 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:52.233527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:52.233531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:52.233690 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:52.234319 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:52.234324 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:52.234468 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 15:21:52.234607 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:21:52.234873 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280}
I20260620 15:21:52.236228 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:52.191379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.0.97.124:35447)
0620 15:21:52.209989 (+ 18610us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.209994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.210014 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:21:52.210058 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.210065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.210140 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.211593 (+ 1453us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.211917 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.213206 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.214933 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:52.215570 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:52.216443 (+ 873us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:52.221005 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:52.221012 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:52.221016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:52.221092 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:52.232308 (+ 11216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.233183 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.233188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.233193 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.233374 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:52.233835 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:52.233840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:52.233844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:52.234179 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:52.235490 (+ 1311us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:52.235614 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 15:21:52.235811 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18458,"thread_start_us":18841,"threads_started":1}
I20260620 15:21:52.238472 22988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5827af20c6dc4d058337bc64e0bdd09f. Latest consensus state: current_term: 1 leader_uuid: "5827af20c6dc4d058337bc64e0bdd09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } } }
I20260620 15:21:52.238790 22988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:52.241804 23023 mvcc.cc:205] Tried to move back new op lower bound from 7298944664420425728 to 7298944664223256576. Current Snapshot: MvccSnapshot[applied={T|T < 7298944664420425728}]
I20260620 15:21:52.243657 22908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3: Bootstrap starting.
I20260620 15:21:52.244664 23026 mvcc.cc:205] Tried to move back new op lower bound from 7298944664420425728 to 7298944664223256576. Current Snapshot: MvccSnapshot[applied={T|T < 7298944664420425728}]
I20260620 15:21:52.247856 22908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:52.249106 22908 log.cc:824] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:52.253947 22908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3: No bootstrap required, opened a new log
I20260620 15:21:52.260794 22988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5827af20c6dc4d058337bc64e0bdd09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } } }
I20260620 15:21:52.261030 22988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:52.264595 22908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } }
I20260620 15:21:52.264959 22908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5491d3eb740649e5b874b4fd99bed5a3, State: Initialized, Role: FOLLOWER
I20260620 15:21:52.265410 22908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } }
I20260620 15:21:52.272971 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5827af20c6dc4d058337bc64e0bdd09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } } }
I20260620 15:21:52.273327 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:52.272320 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } } }
I20260620 15:21:52.273460 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:52.273536 22908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:52.274787 23016 catalog_manager.cc:1347] Generated new cluster ID: 941a63dd1a5d40a58bd3cf3fe2f3f19e
I20260620 15:21:52.274884 23016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:52.275104 22882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:52.278128 22883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5827af20c6dc4d058337bc64e0bdd09f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5491d3eb740649e5b874b4fd99bed5a3"
I20260620 15:21:52.279241 22882 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [term 1 FOLLOWER]: Refusing update from remote peer 5827af20c6dc4d058337bc64e0bdd09f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:52.285403 22883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5827af20c6dc4d058337bc64e0bdd09f in term 1.
I20260620 15:21:52.287257 23002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:52.234884 22767 master_runner.cc:186] Error getting master registration for 127.0.97.125:36561: OK, instance_id { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" instance_seqno: 1781968911464051 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:21:52.292725 22767 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:21:52.302009 23016 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:52.303725 23016 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:52.321765 23032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5827af20c6dc4d058337bc64e0bdd09f. Latest consensus state: current_term: 1 leader_uuid: "5827af20c6dc4d058337bc64e0bdd09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } } }
I20260620 15:21:52.322084 23032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:52.322472 23046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5827af20c6dc4d058337bc64e0bdd09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } } }
I20260620 15:21:52.322796 23046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:52.328620 23016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f: Generated new TSK 0
I20260620 15:21:52.329232 23009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5827af20c6dc4d058337bc64e0bdd09f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5827af20c6dc4d058337bc64e0bdd09f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37069 } } peers { permanent_uuid: "5491d3eb740649e5b874b4fd99bed5a3" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36561 } } peers { permanent_uuid: "03547dce9c95449586d00c8cb1d42e80" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35447 } } }
I20260620 15:21:52.329456 23009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5827af20c6dc4d058337bc64e0bdd09f [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:52.329507 23016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:52.334026 23048 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 15:21:52.406699 22837 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.392951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.0.97.126:37069)
0620 15:21:52.393140 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.393147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.403890 (+ 10743us) server_negotiation.cc:407] Connection header received
0620 15:21:52.405277 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.405284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.405364 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.405441 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.406465 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37266: BlockingRecv error: recv got EOF from 127.0.0.1:37266 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 15:21:52.407718 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.393063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.0.97.125:36561)
0620 15:21:52.393240 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.393246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.405651 (+ 12405us) server_negotiation.cc:407] Connection header received
0620 15:21:52.406277 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.406283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.406369 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.406436 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.407512 (+ 1076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51642: BlockingRecv error: recv got EOF from 127.0.0.1:51642 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:21:52.410013 23021 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.404371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.0.97.124:35447)
0620 15:21:52.404546 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.404552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.408041 (+ 3489us) server_negotiation.cc:407] Connection header received
0620 15:21:52.408628 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.408632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.408706 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.408768 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.409799 (+ 1031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37072: BlockingRecv error: recv got EOF from 127.0.0.1:37072 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 15:21:52.417109 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.415288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.0.97.125:36561)
0620 15:21:52.415439 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.415445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.415743 (+ 298us) server_negotiation.cc:407] Connection header received
0620 15:21:52.415938 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.415943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.416053 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.416165 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.416884 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51652: BlockingRecv error: recv got EOF from 127.0.0.1:51652 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 15:21:52.423557 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.421680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.0.97.125:36561)
0620 15:21:52.421817 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.421822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.422307 (+ 485us) server_negotiation.cc:407] Connection header received
0620 15:21:52.422481 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.422486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.422557 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.422648 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.423392 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51668: BlockingRecv error: recv got EOF from 127.0.0.1:51668 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 15:21:52.430248 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.428416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.0.97.125:36561)
0620 15:21:52.428577 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.428584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.429050 (+ 466us) server_negotiation.cc:407] Connection header received
0620 15:21:52.429203 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.429209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.429288 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.429385 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.430042 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51676: BlockingRecv error: recv got EOF from 127.0.0.1:51676 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 15:21:52.439028 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.437162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.0.97.125:36561)
0620 15:21:52.437351 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.437358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.437835 (+ 477us) server_negotiation.cc:407] Connection header received
0620 15:21:52.438015 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.438020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.438090 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.438192 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.438849 (+ 657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51680: BlockingRecv error: recv got EOF from 127.0.0.1:51680 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 15:21:52.451711 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.449901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.0.97.125:36561)
0620 15:21:52.450060 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.450066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.450499 (+ 433us) server_negotiation.cc:407] Connection header received
0620 15:21:52.450650 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.450655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.450725 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.450836 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.451522 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51690: BlockingRecv error: recv got EOF from 127.0.0.1:51690 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 15:21:52.502085 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.499714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.0.97.125:36561)
0620 15:21:52.499908 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.499917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.500550 (+ 633us) server_negotiation.cc:407] Connection header received
0620 15:21:52.500783 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.500789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.500891 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.500999 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.501871 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51698: BlockingRecv error: recv got EOF from 127.0.0.1:51698 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 15:21:52.548734 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.546488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.0.97.125:36561)
0620 15:21:52.546660 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.546665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.547161 (+ 496us) server_negotiation.cc:407] Connection header received
0620 15:21:52.547364 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.547369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.547460 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.547570 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.548442 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51706: BlockingRecv error: recv got EOF from 127.0.0.1:51706 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 15:21:52.617941 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.616054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.0.97.125:36561)
0620 15:21:52.616283 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.616291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.616644 (+ 353us) server_negotiation.cc:407] Connection header received
0620 15:21:52.616818 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.616824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.616923 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.617044 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.617726 (+ 682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51720: BlockingRecv error: recv got EOF from 127.0.0.1:51720 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 15:21:52.751080 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:52.749217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.0.97.125:36561)
0620 15:21:52.749469 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:52.749476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:52.749718 (+ 242us) server_negotiation.cc:407] Connection header received
0620 15:21:52.749900 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:52.749903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:52.749976 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:52.750060 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:52.750772 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51724: BlockingRecv error: recv got EOF from 127.0.0.1:51724 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260620 15:21:53.012516 22912 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:53.010634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.0.97.125:36561)
0620 15:21:53.010872 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:53.010879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:53.011049 (+ 170us) server_negotiation.cc:407] Connection header received
0620 15:21:53.011311 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:53.011316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:53.011386 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:53.011494 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:53.012279 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51726: BlockingRecv error: recv got EOF from 127.0.0.1:51726 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 15:21:53.044168 22999 catalog_manager.cc:1259] Loaded cluster ID: 941a63dd1a5d40a58bd3cf3fe2f3f19e
I20260620 15:21:53.044282 22999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80: loading cluster ID for follower catalog manager: success
I20260620 15:21:53.047058 22999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80: acquiring CA information for follower catalog manager: success
I20260620 15:21:53.049113 22999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03547dce9c95449586d00c8cb1d42e80: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:21:53.339303 23048 catalog_manager.cc:1259] Loaded cluster ID: 941a63dd1a5d40a58bd3cf3fe2f3f19e
I20260620 15:21:53.339437 23048 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3: loading cluster ID for follower catalog manager: success
I20260620 15:21:53.342129 23048 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3: acquiring CA information for follower catalog manager: success
I20260620 15:21:53.344034 23048 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5491d3eb740649e5b874b4fd99bed5a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:21:53.535532 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:53.528148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.0.97.125:36561)
0620 15:21:53.529306 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:53.529310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:53.529327 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:21:53.529391 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:53.529398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:53.529511 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:53.529611 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:53.530135 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:53.530957 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:53.532938 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:53.533443 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:53.533598 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:53.534173 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:53.534786 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:53.534917 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:53.535107 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 15:21:53.535169 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":911,"threads_started":1}
I20260620 15:21:53.547654 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:53.542218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.0.97.124:35447)
0620 15:21:53.542844 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:53.542852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:53.542871 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:21:53.542974 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:53.542978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:53.543100 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:53.543274 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:53.543648 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:53.544339 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:53.545634 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:53.546019 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:53.546138 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:53.546574 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:53.547002 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:53.547150 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:53.547216 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 15:21:53.547290 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":387,"threads_started":1}
I20260620 15:21:53.574324 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:53.554340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.0.97.126:37069)
0620 15:21:53.566719 (+ 12379us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:53.566725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:53.566746 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:21:53.566857 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:53.566862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:53.567057 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:53.567905 (+ 848us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:53.568082 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:53.569225 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:53.571437 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:53.572100 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:53.572768 (+ 668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:53.572959 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:53.573341 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:53.573854 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:53.573906 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 15:21:53.573989 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12228,"thread_start_us":12875,"threads_started":1}
I20260620 15:21:53.576679 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:53.561321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.0.97.124:35447)
0620 15:21:53.561517 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:53.561525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:53.569811 (+ 8286us) server_negotiation.cc:407] Connection header received
0620 15:21:53.570046 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:53.570051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:53.570133 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:53.570203 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:53.571071 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:53.572122 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:53.574662 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:53.575318 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:53.575408 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:53.575941 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:53.576036 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:53.576133 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:53.576317 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 15:21:53.576401 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:21:53.577764 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:53.554612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.0.97.125:36561)
0620 15:21:53.554817 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:53.554825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:53.569573 (+ 14748us) server_negotiation.cc:407] Connection header received
0620 15:21:53.570027 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:53.570032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:53.570117 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:53.570183 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:53.571073 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:53.572185 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:53.574764 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:53.575437 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:53.575533 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:53.576970 (+ 1437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:53.577098 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:53.577171 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:53.577400 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 15:21:53.577476 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:21:53.579561 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22767
I20260620 15:21:53.580773 22829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:53.690781 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 22767
I20260620 15:21:53.711560 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22841
I20260620 15:21:53.712782 22903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:53.817916 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 22841
I20260620 15:21:53.837730 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 22915
I20260620 15:21:53.838901 22977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:53.943769 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 22915
2026-06-20T15:21:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4705 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:21:54 dist-test-slave-7s64 krb5kdc[23135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:21:54 dist-test-slave-7s64 krb5kdc[23135](info): set up 2 sockets
Jun 20 15:21:54 dist-test-slave-7s64 krb5kdc[23135](info): commencing operation
krb5kdc: starting...
W20260620 15:21:55.045272 389 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/test-user.keytab.
Jun 20 15:21:55 dist-test-slave-7s64 krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:21:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:21:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260620 15:21:55.289906 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35209
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35209,127.0.97.125:45625,127.0.97.124:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:21:55.606377 23148 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:21:55.606665 23148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:55.606735 23148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:55.606833 23148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:55.613848 23148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:55.613922 23148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:55.613977 23148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:55.614025 23148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:55.614076 23148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:55.623129 23148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35209,127.0.97.125:45625,127.0.97.124:40989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:55.624239 23148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:55.625411 23148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:55.636262 23153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:55.636322 23154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:55.636521 23156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:55.637302 23148 server_base.cc:1034] running on GCE node
I20260620 15:21:55.637972 23148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:55.639269 23148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:55.640561 23148 hybrid_clock.cc:648] HybridClock initialized: now 1781968915640461 us; error 48 us; skew 500 ppm
I20260620 15:21:55.640872 23148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:55 dist-test-slave-7s64 krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:55.647104 23148 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260620 15:21:55.648383 23148 webserver.cc:460] Webserver started at http://127.0.97.126:42779/ using document root <none> and password file <none>
I20260620 15:21:55.649101 23148 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:55.649230 23148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:55.649523 23148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:55.652753 23148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/data/instance:
uuid: "5bec28ce99884f0ea390e4c0072597a7"
format_stamp: "Formatted at 2026-06-20 15:21:55 on dist-test-slave-7s64"
server_key: "50c0f7d5f0ed8c7d5fe6327c4fb1032d"
server_key_iv: "e1d5229d3e3ce4648b1b25a8b9d4a481"
server_key_version: "encryptionkey@0"
I20260620 15:21:55.653699 23148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/wal/instance:
uuid: "5bec28ce99884f0ea390e4c0072597a7"
format_stamp: "Formatted at 2026-06-20 15:21:55 on dist-test-slave-7s64"
server_key: "50c0f7d5f0ed8c7d5fe6327c4fb1032d"
server_key_iv: "e1d5229d3e3ce4648b1b25a8b9d4a481"
server_key_version: "encryptionkey@0"
I20260620 15:21:55.658651 23148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 15:21:55.661870 23163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:55.663059 23148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:21:55.663234 23148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/wal
uuid: "5bec28ce99884f0ea390e4c0072597a7"
format_stamp: "Formatted at 2026-06-20 15:21:55 on dist-test-slave-7s64"
server_key: "50c0f7d5f0ed8c7d5fe6327c4fb1032d"
server_key_iv: "e1d5229d3e3ce4648b1b25a8b9d4a481"
server_key_version: "encryptionkey@0"
I20260620 15:21:55.663444 23148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:55.674049 23148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:55.678087 23148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:55.678340 23148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:55.695839 23148 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35209
I20260620 15:21:55.695854 23214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35209 every 8 connection(s)
I20260620 15:21:55.697078 23148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/data/info.pb
I20260620 15:21:55.699771 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 23148
I20260620 15:21:55.699976 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-0/wal/instance
I20260620 15:21:55.700443 389 external_mini_cluster.cc:1428] Setting key 7aeaddffdac7a65775cc1856659b2907
I20260620 15:21:55.700546 23215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:55.706949 23215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:21:55 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.125@KRBTEST.COM" created.
I20260620 15:21:55.730846 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:55.709856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35209 (local address 127.0.0.1:37936)
0620 15:21:55.711022 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:55.711061 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:55.712751 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:55.713630 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:55.713635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:55.714151 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:55.714468 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:55.714483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:55.716780 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:55.716784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:55.717840 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:55.717851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:55.719188 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:55.719193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:55.719310 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:55.719981 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:55.720007 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:55.722510 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:55.726245 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:55.726259 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:55.726272 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:55.726851 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:55.727286 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:55.727292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:55.727295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:55.727442 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:55.728195 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:55.728203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:55.728437 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 15:21:55.728788 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 15:21:55.729448 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":185,"spinlock_wait_cycles":16256,"thread_start_us":704,"threads_started":1}
I20260620 15:21:55.730867 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:55.709856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.0.97.126:35209)
0620 15:21:55.710647 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:55.710664 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:55.711196 (+ 532us) server_negotiation.cc:407] Connection header received
0620 15:21:55.713095 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:55.713105 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:55.713393 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:55.713608 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:55.714686 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:55.716634 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:55.717980 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:55.718972 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:55.719160 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:55.722761 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:55.722773 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:55.722788 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:55.722823 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:55.726076 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:55.726993 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:55.727003 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:55.727009 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:55.727155 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:55.727562 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:55.727570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:55.727574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:55.728050 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:55.728283 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:55.728909 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 15:21:55.729327 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":552,"thread_start_us":331,"threads_started":1}
I20260620 15:21:55.734012 23215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:21:55.735091 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:55.734714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56148)
0620 15:21:55.734883 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:55.734983 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 15:21:55.735324 23164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
I20260620 15:21:55.737046 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:55.736567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56160)
0620 15:21:55.736821 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:55.736908 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:21:55.737289 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:45625: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab.
I20260620 15:21:55.755405 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:45625
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35209,127.0.97.125:45625,127.0.97.124:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:21:55.786598 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } attempt: 1
I20260620 15:21:55.787694 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:55.787275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56170)
0620 15:21:55.787448 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:55.787551 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 15:21:55.788774 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:55.788330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56182)
0620 15:21:55.788574 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:55.788643 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:21:55.788933 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:45625: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
I20260620 15:21:55.857137 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } attempt: 2
I20260620 15:21:55.858338 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:55.857870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56198)
0620 15:21:55.858092 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:55.858191 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 15:21:55.859344 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:55.858987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56210)
0620 15:21:55.859154 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:55.859223 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:21:55.859512 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:45625: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
I20260620 15:21:55.950742 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } attempt: 3
I20260620 15:21:55.952071 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:55.951523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56216)
0620 15:21:55.951748 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:55.951854 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 15:21:55.953152 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:55.952796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56222)
0620 15:21:55.952951 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:55.953026 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 15:21:55.953329 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:45625: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
W20260620 15:21:56.064217 23222 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:21:56.064606 23222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:56.064725 23222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:56.064868 23222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:21:56.071779 23222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:56.071889 23222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:56.071993 23222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:56.072086 23222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:56.072175 23222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:56.081990 23222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35209,127.0.97.125:45625,127.0.97.124:40989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:56.083245 23222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:56.084565 23222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:56.095989 23227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:56.096043 23228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:56.096364 23230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:56.096570 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } attempt: 4
I20260620 15:21:56.097388 23222 server_base.cc:1034] running on GCE node
I20260620 15:21:56.097834 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.097204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56234)
0620 15:21:56.097559 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.097666 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260620 15:21:56.098213 23222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:56.098829 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.098489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56238)
0620 15:21:56.098616 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.098686 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:21:56.099018 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:45625: Network error: Client connection negotiation failed: client connection to 127.0.97.125:45625: connect: Connection refused (error 111)
I20260620 15:21:56.099604 23222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:56.100847 23222 hybrid_clock.cc:648] HybridClock initialized: now 1781968916100785 us; error 30 us; skew 500 ppm
I20260620 15:21:56.101133 23222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:56.104215 23222 init.cc:377] Logged in from keytab as kudu/127.0.97.125@KRBTEST.COM (short username kudu)
I20260620 15:21:56.105813 23222 webserver.cc:460] Webserver started at http://127.0.97.125:42293/ using document root <none> and password file <none>
I20260620 15:21:56.106500 23222 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:56.106642 23222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:56.106969 23222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:56.110015 23222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/data/instance:
uuid: "fc7319eb3a0f477aba5b8a5bde4a304d"
format_stamp: "Formatted at 2026-06-20 15:21:56 on dist-test-slave-7s64"
server_key: "6d962f124a79b71869923deaac2b4ef9"
server_key_iv: "24fc13f80b2c067bde6ab017e69cd6d9"
server_key_version: "encryptionkey@0"
I20260620 15:21:56.110834 23222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/wal/instance:
uuid: "fc7319eb3a0f477aba5b8a5bde4a304d"
format_stamp: "Formatted at 2026-06-20 15:21:56 on dist-test-slave-7s64"
server_key: "6d962f124a79b71869923deaac2b4ef9"
server_key_iv: "24fc13f80b2c067bde6ab017e69cd6d9"
server_key_version: "encryptionkey@0"
I20260620 15:21:56.115952 23222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 15:21:56.119328 23237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:56.120759 23222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:21:56.120935 23222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/wal
uuid: "fc7319eb3a0f477aba5b8a5bde4a304d"
format_stamp: "Formatted at 2026-06-20 15:21:56 on dist-test-slave-7s64"
server_key: "6d962f124a79b71869923deaac2b4ef9"
server_key_iv: "24fc13f80b2c067bde6ab017e69cd6d9"
server_key_version: "encryptionkey@0"
I20260620 15:21:56.121134 23222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:56.132428 23222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:56.136191 23222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:56.136437 23222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:56.154373 23222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:45625
I20260620 15:21:56.154414 23288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:45625 every 8 connection(s)
I20260620 15:21:56.155591 23222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/data/info.pb
I20260620 15:21:56.159076 23289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:56.165122 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 23222
I20260620 15:21:56.165328 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-1/wal/instance
I20260620 15:21:56.165884 389 external_mini_cluster.cc:1428] Setting key 47bc053860539d3243b817c0860164d3
I20260620 15:21:56.165594 23289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:56.188738 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.168661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.0.97.126:35209)
0620 15:21:56.168867 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.168873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.170104 (+ 1231us) server_negotiation.cc:407] Connection header received
0620 15:21:56.171902 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.171910 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.172009 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.172135 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.173622 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.174547 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.177364 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.177809 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.177927 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.182327 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.182332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.182336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.182370 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.185374 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.186173 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.186177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.186179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.186307 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.186775 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.186781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.186784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.187082 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.187261 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.188064 (+ 803us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.188343 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:21:56.189661 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.169155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35209 (local address 127.0.0.1:37952)
0620 15:21:56.169919 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.169964 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:56.171589 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:56.172337 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:56.172345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:56.172916 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:56.173433 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.173449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.174752 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.174762 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.177154 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.177173 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.178069 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.178079 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.178434 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.179200 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:56.179246 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:56.182083 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:56.185530 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.185540 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.185556 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.186016 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.186427 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.186433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.186436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.186631 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.187231 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:56.187240 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:56.187613 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 15:21:56.187955 (+ 342us) client_negotiation.cc:240] Negotiation successful
0620 15:21:56.188457 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":335,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.124@KRBTEST.COM; defaulting to no policy
I20260620 15:21:56.191439 23289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.124@KRBTEST.COM" created.
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260620 15:21:56.206964 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.192349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56244)
0620 15:21:56.192611 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.192627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:56.192712 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:56.193734 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:56.193738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:56.194158 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:56.194280 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.194288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.195683 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.195691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.197232 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.197250 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.198024 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.198035 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.198261 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.198843 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:56.198871 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:56.201604 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:56.204648 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.204655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.204659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.204982 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.205344 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.205349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.205352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.205486 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.206048 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:56.206053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:56.206206 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 15:21:56.206344 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 15:21:56.206589 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 15:21:56.207046 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.192647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.0.97.125:45625)
0620 15:21:56.193316 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.193321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.193344 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:21:56.193382 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.193387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.193597 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.193743 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.194452 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.195510 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.197367 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.197866 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.197959 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.201798 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.201807 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.201818 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.201849 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.204525 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.205105 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.205118 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.205121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.205220 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.205616 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.205621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.205624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.205926 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.206128 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.206508 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.206700 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":369,"threads_started":1}
I20260620 15:21:56.209244 23289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:21:56.210228 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.209809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46788)
0620 15:21:56.209997 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.210103 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:21:56.210459 23240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
I20260620 15:21:56.211894 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.211589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46804)
0620 15:21:56.211720 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.211799 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 15:21:56.212117 23289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:40989: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab.
I20260620 15:21:56.221817 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
/tmp/dist-test-taskRnfwZV/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:40989
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35209,127.0.97.125:45625,127.0.97.124:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:21:56.261368 23289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } attempt: 1
I20260620 15:21:56.262414 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.262012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46814)
0620 15:21:56.262183 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.262277 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 15:21:56.263346 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.263002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46830)
0620 15:21:56.263167 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.263237 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:21:56.263552 23289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:40989: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
I20260620 15:21:56.331784 23289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } attempt: 2
I20260620 15:21:56.332994 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.332465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46840)
0620 15:21:56.332712 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.332835 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 15:21:56.334010 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.333667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46842)
0620 15:21:56.333814 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.333892 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:21:56.334210 23289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:40989: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
I20260620 15:21:56.398288 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } attempt: 5
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260620 15:21:56.411899 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.399081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56258)
0620 15:21:56.399305 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.399326 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:56.399473 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:56.399860 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:56.399867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:56.400252 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:56.400453 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.400463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.401511 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.401519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.402389 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.402401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.403111 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.403134 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.403326 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.403956 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:56.403986 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:56.406023 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:56.409646 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.409649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.409652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.410010 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.410449 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.410452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.410454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.410588 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.411210 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:56.411214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:56.411336 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 15:21:56.411470 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 15:21:56.411651 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:21:56.412211 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.399126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.0.97.125:45625)
0620 15:21:56.399343 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.399351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.399396 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:21:56.399622 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.399631 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.399711 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.399818 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.400654 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.401370 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.402538 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.402954 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.403091 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.406222 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.406228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.406233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.406279 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.409526 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.410133 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.410140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.410145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.410334 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.410701 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.410707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.410710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.411098 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.411243 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.411590 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.411806 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 15:21:56.413311 23215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:21:56.414284 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.413924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46858)
0620 15:21:56.414075 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.414145 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 15:21:56.415293 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.414949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46870)
0620 15:21:56.415086 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.415211 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:21:56.415565 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:40989: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
I20260620 15:21:56.425401 23289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } attempt: 3
I20260620 15:21:56.426332 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.425919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46874)
0620 15:21:56.426093 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.426204 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 15:21:56.427354 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.426939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46890)
0620 15:21:56.427149 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.427218 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260620 15:21:56.427527 23289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:40989: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
I20260620 15:21:56.466794 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } attempt: 1
I20260620 15:21:56.467902 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.467476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46900)
0620 15:21:56.467676 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.467756 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:21:56.468989 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.468652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46914)
0620 15:21:56.468823 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.468889 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 15:21:56.469192 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:40989: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
W20260620 15:21:56.532956 23296 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:21:56.533303 23296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:21:56.533445 23296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:21:56.533578 23296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 15:21:56.537388 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } attempt: 2
I20260620 15:21:56.538468 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.538004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46926)
0620 15:21:56.538236 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.538318 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 15:21:56.539410 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.539097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46936)
0620 15:21:56.539238 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.539314 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 15:21:56.539608 23215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:40989: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
W20260620 15:21:56.541041 23296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:21:56.541147 23296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:21:56.541271 23296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:21:56.541354 23296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:21:56.541460 23296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:21:56.550873 23296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35209,127.0.97.125:45625,127.0.97.124:40989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 14:18:34 UTC on ccycloud.quasar-gzvchx.root.comops.site
build id 4457506
ASAN enabled
I20260620 15:21:56.552059 23296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:21:56.553390 23296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:21:56.564152 23304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:56.564150 23301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:21:56.564755 23302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:21:56.565824 23296 server_base.cc:1034] running on GCE node
I20260620 15:21:56.566581 23296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:21:56.567900 23296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:21:56.569097 23296 hybrid_clock.cc:648] HybridClock initialized: now 1781968916569035 us; error 41 us; skew 500 ppm
I20260620 15:21:56.569552 23296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:21:56.570725 23289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } attempt: 4
I20260620 15:21:56.571767 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.571325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46948)
0620 15:21:56.571545 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.571622 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:21:56.572759 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.572376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46956)
0620 15:21:56.572560 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.572653 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 15:21:56.572955 23289 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:40989: Network error: Client connection negotiation failed: client connection to 127.0.97.124:40989: connect: Connection refused (error 111)
I20260620 15:21:56.573218 23296 init.cc:377] Logged in from keytab as kudu/127.0.97.124@KRBTEST.COM (short username kudu)
I20260620 15:21:56.574615 23296 webserver.cc:460] Webserver started at http://127.0.97.124:36603/ using document root <none> and password file <none>
I20260620 15:21:56.575253 23296 fs_manager.cc:359] Metadata directory not provided
I20260620 15:21:56.575357 23296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:21:56.575677 23296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:21:56.578552 23296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/data/instance:
uuid: "6d8ad47ff6b04c4f9b68ed302659c46b"
format_stamp: "Formatted at 2026-06-20 15:21:56 on dist-test-slave-7s64"
server_key: "e8a90da16742e92381c96b726030e60a"
server_key_iv: "16d04d1a571b48981024c147ff1c635f"
server_key_version: "encryptionkey@0"
I20260620 15:21:56.579396 23296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/wal/instance:
uuid: "6d8ad47ff6b04c4f9b68ed302659c46b"
format_stamp: "Formatted at 2026-06-20 15:21:56 on dist-test-slave-7s64"
server_key: "e8a90da16742e92381c96b726030e60a"
server_key_iv: "16d04d1a571b48981024c147ff1c635f"
server_key_version: "encryptionkey@0"
I20260620 15:21:56.584187 23296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260620 15:21:56.587271 23311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:21:56.588732 23296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 15:21:56.588884 23296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/wal
uuid: "6d8ad47ff6b04c4f9b68ed302659c46b"
format_stamp: "Formatted at 2026-06-20 15:21:56 on dist-test-slave-7s64"
server_key: "e8a90da16742e92381c96b726030e60a"
server_key_iv: "16d04d1a571b48981024c147ff1c635f"
server_key_version: "encryptionkey@0"
I20260620 15:21:56.589083 23296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:21:56.605324 23296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:21:56.609336 23296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:21:56.609619 23296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:21:56.628329 23296 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:40989
I20260620 15:21:56.628355 23362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:40989 every 8 connection(s)
I20260620 15:21:56.629837 23296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/data/info.pb
I20260620 15:21:56.631737 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu as pid 23296
I20260620 15:21:56.631927 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskRnfwZV/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781968566423581-389-0/minicluster-data/master-2/wal/instance
I20260620 15:21:56.632510 389 external_mini_cluster.cc:1428] Setting key c283278b4d68c309abe341584a1acc20
I20260620 15:21:56.633929 23363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:21:56.642455 23363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:21:56.645804 23215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } attempt: 3
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260620 15:21:56.655469 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.634212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.0.97.124:40989)
0620 15:21:56.634930 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.634941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.634979 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:21:56.636459 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.636479 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.636907 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.637212 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.638834 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.640165 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.641947 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.642586 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.642747 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.647577 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.647589 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.647607 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.647656 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.651024 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.651768 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.651777 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.651785 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.651993 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.652488 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.652493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.652496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.653164 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.653478 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.653920 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.654339 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":320,"threads_started":1}
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:56.664350 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.645310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35209 (local address 127.0.0.1:37968)
0620 15:21:56.646039 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.646083 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:56.646278 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:56.646836 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:56.646843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:56.647995 (+ 1152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:56.648358 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.648375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.649372 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.649376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.651291 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.651316 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.652438 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.652447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.652617 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.653270 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:56.653296 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:56.657788 (+ 4492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:56.661578 (+ 3790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.661588 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.661595 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.661974 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.662352 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.662357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.662361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.662512 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.663283 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:56.663290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:56.663451 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 15:21:56.663639 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 15:21:56.663922 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":359,"threads_started":1}
I20260620 15:21:56.664434 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.645469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.0.97.126:35209)
0620 15:21:56.645678 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.645685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.646162 (+ 477us) server_negotiation.cc:407] Connection header received
0620 15:21:56.646428 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.646434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.646515 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.646588 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.648519 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.649199 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.651451 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.652036 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.652169 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.657980 (+ 5811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.657987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.657991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.658026 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.661471 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.662092 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.662099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.662102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.662241 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.662652 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.662657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.662661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.663144 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.663290 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.663755 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.664073 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":141,"server-negotiator.queue_time_us":75}
I20260620 15:21:56.664811 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.646296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46976)
0620 15:21:56.646466 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.646484 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:56.646580 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:56.647688 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:56.647695 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:56.648114 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:56.648240 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.648248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.649985 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.649990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.651358 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.651375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.652178 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.652182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.652460 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.653052 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:56.653074 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:56.657421 (+ 4347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:56.661538 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.661543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.661547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.662066 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.662602 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.662609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.662614 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.662788 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.663372 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:56.663377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:56.663547 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 15:21:56.663707 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 15:21:56.663964 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":49}
I20260620 15:21:56.664849 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.646788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.0.97.124:40989)
0620 15:21:56.647408 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.647413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.647443 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:21:56.647482 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.647485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.647552 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.647664 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.648414 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.649846 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.651486 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.651997 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.652258 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.657693 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.657725 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.657730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.657765 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.661381 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.662239 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.662246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.662250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.662456 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.662965 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.662970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.662973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.663283 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.663404 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.664230 (+ 826us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.664432 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"spinlock_wait_cycles":16128,"thread_start_us":291,"threads_started":1}
I20260620 15:21:56.666433 23363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260620 15:21:56.670110 23215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7: Bootstrap starting.
I20260620 15:21:56.674223 23215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7: Neither blocks nor log segments found. Creating new log.
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260620 15:21:56.675384 23215 log.cc:824] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:56.677987 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.659157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.0.97.126:35209)
0620 15:21:56.661001 (+ 1844us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.661007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.661032 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:21:56.661078 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.661083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.661150 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.661270 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.662771 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.664030 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.665802 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.666422 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.666550 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.671104 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.671111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.671115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.671147 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.674775 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.675616 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.675648 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.675654 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.675776 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.676141 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.676147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.676150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.676580 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.676855 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.677158 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.677392 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1692,"thread_start_us":1460,"threads_started":1}
I20260620 15:21:56.683568 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.668679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56262 (local address 127.0.97.125:45625)
0620 15:21:56.668882 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.668888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.668907 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:21:56.668965 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.668970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.669041 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.669137 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.669943 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.670730 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.671832 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.672378 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.672546 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.675569 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.675576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.675580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.675618 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.679838 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.680843 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.680852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.680858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.681058 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.681543 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.681550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.681555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.681939 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.682102 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.682757 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.683042 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:21:56.684082 23215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7: No bootstrap required, opened a new log
I20260620 15:21:56.684916 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.667320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:45625 (local address 127.0.0.1:56262)
0620 15:21:56.667559 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.667578 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:56.667663 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:56.669343 (+ 1680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:56.669350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:56.669679 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:56.669801 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.669808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.670863 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.670871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.671697 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.671708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.672555 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.672560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.672748 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.673379 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:56.673394 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:56.675375 (+ 1981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:56.679976 (+ 4601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.679984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.679990 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.680703 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.681192 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.681200 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.681206 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.681405 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.682080 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:56.682087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:56.682285 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 15:21:56.682456 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 15:21:56.684434 (+ 1978us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260620 15:21:56.689422 23363 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:21:56.693257 23215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } }
I20260620 15:21:56.693605 23215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bec28ce99884f0ea390e4c0072597a7, State: Initialized, Role: FOLLOWER
I20260620 15:21:56.694082 23215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } }
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260620 15:21:56.695015 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } } }
I20260620 15:21:56.695508 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260620 15:21:56.698199 23215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:56.703846 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.679456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56266 (local address 127.0.97.125:45625)
0620 15:21:56.683899 (+ 4443us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.683906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.683927 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:21:56.683981 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.683989 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.684071 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.686982 (+ 2911us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.687150 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.688612 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.690289 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.690750 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.690883 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.696038 (+ 5155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.696044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.696049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.696083 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.699910 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.700884 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.700889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.700893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.701057 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.701480 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.701485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.701489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.701849 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.702099 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.703240 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.703450 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4314,"thread_start_us":9185,"threads_started":1}
I20260620 15:21:56.705662 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.690022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:46978)
0620 15:21:56.690192 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.690209 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:56.690300 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:56.690866 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:56.690869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:56.691159 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:56.691281 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.691288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.692626 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.692633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.693897 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.693911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.695025 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.695032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.695256 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.695822 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:56.695848 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:56.698083 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:56.702534 (+ 4451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.702542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.702548 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.703127 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.703580 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.703587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.703593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.703744 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.704467 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:56.704474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:56.704658 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 15:21:56.704810 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 15:21:56.705160 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 15:21:56.706969 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.690290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.0.97.124:40989)
0620 15:21:56.690438 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.690442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.690454 (+ 12us) server_negotiation.cc:407] Connection header received
0620 15:21:56.690526 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.690530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.690602 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.690696 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.691416 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.692464 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.694051 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.694726 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.694847 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.698291 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.698299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.698307 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.698347 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.702375 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.703299 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.703306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.703311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.703459 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.703881 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.703887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.703892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.704325 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.706047 (+ 1722us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.706234 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.706513 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 15:21:56.712082 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:21:56.713263 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b: Bootstrap starting.
I20260620 15:21:56.714515 23190 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5bec28ce99884f0ea390e4c0072597a7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37978
I20260620 15:21:56.714875 23190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 15:21:56.715072 23190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:56.717849 23363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:56.717841 23190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } }
I20260620 15:21:56.719381 23363 log.cc:824] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:56.719919 23190 leader_election.cc:288] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [CANDIDATE]: Term 1 election: Requested vote from peers fc7319eb3a0f477aba5b8a5bde4a304d (127.0.97.125:45625), 6d8ad47ff6b04c4f9b68ed302659c46b (127.0.97.124:40989)
I20260620 15:21:56.723332 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b: No bootstrap required, opened a new log
I20260620 15:21:56.737041 23363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } }
I20260620 15:21:56.737547 23363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d8ad47ff6b04c4f9b68ed302659c46b, State: Initialized, Role: FOLLOWER
I20260620 15:21:56.738067 23363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } }
I20260620 15:21:56.744019 23389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } } }
I20260620 15:21:56.746404 23363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:56.746603 23338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bec28ce99884f0ea390e4c0072597a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b"
I20260620 15:21:56.747136 23338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:56.747996 23389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:56.750916 23338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bec28ce99884f0ea390e4c0072597a7 in term 1.
I20260620 15:21:56.751622 23166 leader_election.cc:302] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bec28ce99884f0ea390e4c0072597a7, 6d8ad47ff6b04c4f9b68ed302659c46b; no voters:
I20260620 15:21:56.752221 23372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:21:56.757119 23372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [term 1 LEADER]: Becoming Leader. State: Replica: 5bec28ce99884f0ea390e4c0072597a7, State: Running, Role: LEADER
I20260620 15:21:56.757791 23372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } }
I20260620 15:21:56.764545 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.724421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56282 (local address 127.0.97.125:45625)
0620 15:21:56.724648 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.724655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.725134 (+ 479us) server_negotiation.cc:407] Connection header received
0620 15:21:56.725806 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.725815 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.725922 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.726016 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.734451 (+ 8435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.735983 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.740547 (+ 4564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.741299 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.741450 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.745401 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.745407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.745413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.745457 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.756904 (+ 11447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.761508 (+ 4604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.761517 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.761528 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.761753 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.762210 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.762216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.762219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.762580 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.762895 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.763872 (+ 977us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.764133 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 15:21:56.766692 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.732344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.0.97.124:40989)
0620 15:21:56.732572 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.732581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.739326 (+ 6745us) server_negotiation.cc:407] Connection header received
0620 15:21:56.739557 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.739564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.739641 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.739743 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.742756 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.743918 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.751943 (+ 8025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.752880 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.752999 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.758300 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.758307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.758312 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.758352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.763282 (+ 4930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.764173 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.764178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.764181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.764367 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.764944 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.764949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.764952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.765285 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.765549 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.766055 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.766325 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:21:56.766925 23400 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:21:56.767014 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bec28ce99884f0ea390e4c0072597a7. Latest consensus state: current_term: 1 leader_uuid: "5bec28ce99884f0ea390e4c0072597a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } } }
I20260620 15:21:56.767458 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:56.768819 23401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:21:56.770161 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.724249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.0.97.126:35209)
0620 15:21:56.724470 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.724479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.732065 (+ 7586us) server_negotiation.cc:407] Connection header received
0620 15:21:56.733128 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.733134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.733205 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.733269 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.737769 (+ 4500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.738833 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.745381 (+ 6548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.746159 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.746275 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.749809 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.749819 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.749827 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.749872 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.754967 (+ 5095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.767490 (+ 12523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.767499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.767505 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.767911 (+ 406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.768345 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.768350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.768354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.768751 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.769046 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.769556 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.769804 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 15:21:56.776391 23401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:21:56.772658 23148 master_runner.cc:186] Error getting master registration for 127.0.97.125:45625: OK, instance_id { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" instance_seqno: 1781968916149192 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:21:56.785885 23148 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:21:56.790709 23338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [term 1 FOLLOWER]: Refusing update from remote peer 5bec28ce99884f0ea390e4c0072597a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:56.791741 23393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:21:56.811738 23408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:21:56.811887 23408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:21:56.793999 23296 master_runner.cc:186] Error getting master registration for 127.0.97.125:45625: OK, instance_id { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" instance_seqno: 1781968916149192 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:21:56.815327 23296 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:21:56.823964 23389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bec28ce99884f0ea390e4c0072597a7. Latest consensus state: current_term: 1 leader_uuid: "5bec28ce99884f0ea390e4c0072597a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } } }
I20260620 15:21:56.824218 23389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:56.828474 23407 mvcc.cc:205] Tried to move back new op lower bound from 7298944683164991488 to 7298944683045855232. Current Snapshot: MvccSnapshot[applied={T|T < 7298944683164991488}]
I20260620 15:21:56.830076 23410 mvcc.cc:205] Tried to move back new op lower bound from 7298944683164991488 to 7298944683045855232. Current Snapshot: MvccSnapshot[applied={T|T < 7298944683164991488}]
I20260620 15:21:56.836817 23389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bec28ce99884f0ea390e4c0072597a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } } }
I20260620 15:21:56.837024 23389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:56.837078 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bec28ce99884f0ea390e4c0072597a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } } }
I20260620 15:21:56.837698 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:21:56.838792 23401 catalog_manager.cc:1347] Generated new cluster ID: 5389a5241f3b4c6fbb3ad8be3049e0a5
I20260620 15:21:56.838897 23401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:21:56.860442 23401 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:21:56.862383 23401 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:21:56.872174 23289 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } attempt: 5
Jun 20 15:21:56 dist-test-slave-7s64 krb5kdc[23135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781968916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260620 15:21:56.884229 23401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7: Generated new TSK 0
I20260620 15:21:56.885777 23401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:21:56.892097 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.872764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:40989 (local address 127.0.0.1:47004)
0620 15:21:56.873090 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 15:21:56.873109 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:21:56.873242 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:21:56.874270 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:21:56.874276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:21:56.874798 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:21:56.875000 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.875012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.876545 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.876551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.878021 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:21:56.878036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.880874 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.880881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:21:56.881089 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.881759 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:21:56.881803 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:21:56.884611 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:21:56.888945 (+ 4334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.888951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.888955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.889461 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.889873 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:21:56.889878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:21:56.889882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:21:56.890040 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:21:56.891120 (+ 1080us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:21:56.891125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:21:56.891274 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 15:21:56.891396 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 15:21:56.891706 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260620 15:21:56.894246 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:56.873581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.0.97.124:40989)
0620 15:21:56.873767 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:56.873773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:56.873790 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:21:56.873841 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:56.873846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:56.873915 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:56.874007 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:56.875180 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.876360 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.879673 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:56.880544 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:56.880670 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:56.884767 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:21:56.884773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:21:56.884777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:21:56.884813 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:21:56.888821 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.889584 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.889590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.889594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.889767 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:21:56.890232 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:21:56.890240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:21:56.890243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:21:56.890914 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:21:56.892883 (+ 1969us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:56.893112 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 15:21:56.893518 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:21:56.905861 23289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d: Bootstrap starting.
I20260620 15:21:56.909950 23289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d: Neither blocks nor log segments found. Creating new log.
I20260620 15:21:56.912151 23289 log.cc:824] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:21:56.928869 23289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d: No bootstrap required, opened a new log
I20260620 15:21:56.943818 23289 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } }
I20260620 15:21:56.944252 23289 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7319eb3a0f477aba5b8a5bde4a304d, State: Initialized, Role: FOLLOWER
I20260620 15:21:56.944783 23289 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } }
I20260620 15:21:56.951685 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } } }
I20260620 15:21:56.952068 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:56.953787 23289 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:21:56.954223 23263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:21:56.957899 23263 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [term 1 FOLLOWER]: Refusing update from remote peer 5bec28ce99884f0ea390e4c0072597a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:21:56.958628 23264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bec28ce99884f0ea390e4c0072597a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d"
I20260620 15:21:56.962049 23264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bec28ce99884f0ea390e4c0072597a7 in term 1.
I20260620 15:21:56.963198 23393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:21:57.001470 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bec28ce99884f0ea390e4c0072597a7. Latest consensus state: current_term: 1 leader_uuid: "5bec28ce99884f0ea390e4c0072597a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } } }
I20260620 15:21:57.001811 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:21:57.001554 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5bec28ce99884f0ea390e4c0072597a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } } }
I20260620 15:21:57.002173 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d [sys.catalog]: This master's current role is: FOLLOWER
W20260620 15:21:57.004673 389 mini_kdc.cc:348] Time spent kdestroy: real 0.108s user 0.009s sys 0.009s
I20260620 15:21:57.029167 23435 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 15:21:57.041584 23369 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.038594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.0.97.126:35209)
0620 15:21:57.038822 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.038828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.039686 (+ 858us) server_negotiation.cc:407] Connection header received
0620 15:21:57.039971 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.039977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.040056 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.040134 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.041315 (+ 1181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38006: BlockingRecv error: recv got EOF from 127.0.0.1:38006 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 15:21:57.043484 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.040048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.0.97.125:45625)
0620 15:21:57.040232 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.040237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.042040 (+ 1803us) server_negotiation.cc:407] Connection header received
0620 15:21:57.042252 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.042258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.042331 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.042395 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.043289 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56286: BlockingRecv error: recv got EOF from 127.0.0.1:56286 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 15:21:57.048246 23368 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.045824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47014 (local address 127.0.97.124:40989)
0620 15:21:57.046016 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.046022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.046589 (+ 567us) server_negotiation.cc:407] Connection header received
0620 15:21:57.046711 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.046716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.046806 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.046904 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.047973 (+ 1069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47014: BlockingRecv error: recv got EOF from 127.0.0.1:47014 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 15:21:57.058264 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.052790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.0.97.125:45625)
0620 15:21:57.052968 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.052973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.056943 (+ 3970us) server_negotiation.cc:407] Connection header received
0620 15:21:57.057136 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.057141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.057245 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.057349 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.058007 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56294: BlockingRecv error: recv got EOF from 127.0.0.1:56294 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 15:21:57.066165 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.062400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.0.97.125:45625)
0620 15:21:57.062542 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.062548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.065021 (+ 2473us) server_negotiation.cc:407] Connection header received
0620 15:21:57.065152 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.065157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.065227 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.065323 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.065993 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56306: BlockingRecv error: recv got EOF from 127.0.0.1:56306 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 15:21:57.072480 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.070710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.0.97.125:45625)
0620 15:21:57.070845 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.070850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.071279 (+ 429us) server_negotiation.cc:407] Connection header received
0620 15:21:57.071447 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.071452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.071531 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.071627 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.072286 (+ 659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56322: BlockingRecv error: recv got EOF from 127.0.0.1:56322 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 15:21:57.082186 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.079476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.0.97.125:45625)
0620 15:21:57.079659 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.079674 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.081013 (+ 1339us) server_negotiation.cc:407] Connection header received
0620 15:21:57.081169 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.081174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.081245 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.081339 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.081999 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56324: BlockingRecv error: recv got EOF from 127.0.0.1:56324 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260620 15:21:57.095244 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.093395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.0.97.125:45625)
0620 15:21:57.093530 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.093535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.093986 (+ 451us) server_negotiation.cc:407] Connection header received
0620 15:21:57.094152 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.094157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.094230 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.094332 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.095051 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56338: BlockingRecv error: recv got EOF from 127.0.0.1:56338 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 15:21:57.116375 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.114302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.0.97.125:45625)
0620 15:21:57.114470 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.114476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.114984 (+ 508us) server_negotiation.cc:407] Connection header received
0620 15:21:57.115165 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.115172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.115271 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.115386 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.116155 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56346: BlockingRecv error: recv got EOF from 127.0.0.1:56346 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 15:21:57.162160 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.160480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.0.97.125:45625)
0620 15:21:57.160700 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.160708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.160730 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:21:57.160814 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.160818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.160911 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.161029 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.161929 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56358: BlockingRecv error: recv got EOF from 127.0.0.1:56358 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 15:21:57.232754 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.230393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.0.97.125:45625)
0620 15:21:57.230599 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.230607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.231148 (+ 541us) server_negotiation.cc:407] Connection header received
0620 15:21:57.231371 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.231377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.231474 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.231592 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.232523 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56360: BlockingRecv error: recv got EOF from 127.0.0.1:56360 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 15:21:57.366050 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.364121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.0.97.125:45625)
0620 15:21:57.364359 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.364372 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.364533 (+ 161us) server_negotiation.cc:407] Connection header received
0620 15:21:57.364789 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.364793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.364879 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.364966 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.365796 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56376: BlockingRecv error: recv got EOF from 127.0.0.1:56376 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 15:21:57.556407 23393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5bec28ce99884f0ea390e4c0072597a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bec28ce99884f0ea390e4c0072597a7" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35209 } } peers { permanent_uuid: "fc7319eb3a0f477aba5b8a5bde4a304d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 45625 } } peers { permanent_uuid: "6d8ad47ff6b04c4f9b68ed302659c46b" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 40989 } } }
I20260620 15:21:57.556763 23393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bec28ce99884f0ea390e4c0072597a7 [sys.catalog]: This master's current role is: LEADER
W20260620 15:21:57.627866 23293 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:21:57.625944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.0.97.125:45625)
0620 15:21:57.626174 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:57.626180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:57.626449 (+ 269us) server_negotiation.cc:407] Connection header received
0620 15:21:57.626672 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:57.626678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:57.626770 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:57.626879 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:21:57.627592 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56390: BlockingRecv error: recv got EOF from 127.0.0.1:56390 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:21:57.813653 23408 catalog_manager.cc:1259] Loaded cluster ID: 5389a5241f3b4c6fbb3ad8be3049e0a5
I20260620 15:21:57.813772 23408 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b: loading cluster ID for follower catalog manager: success
I20260620 15:21:57.816341 23408 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b: acquiring CA information for follower catalog manager: success
I20260620 15:21:57.818082 23408 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d8ad47ff6b04c4f9b68ed302659c46b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:21:58.033296 23435 catalog_manager.cc:1259] Loaded cluster ID: 5389a5241f3b4c6fbb3ad8be3049e0a5
I20260620 15:21:58.033408 23435 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d: loading cluster ID for follower catalog manager: success
I20260620 15:21:58.035858 23435 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d: acquiring CA information for follower catalog manager: success
I20260620 15:21:58.037725 23435 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc7319eb3a0f477aba5b8a5bde4a304d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:21:58.155435 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:58.148836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.0.97.125:45625)
0620 15:21:58.149663 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:58.149673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:58.149700 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:21:58.149764 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:58.149768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:58.149917 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:58.150057 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:58.150517 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:58.151289 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:58.152652 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:58.153109 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:58.153278 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:58.153772 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:58.154693 (+ 921us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:58.154862 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:58.155036 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 15:21:58.155118 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":506,"threads_started":1}
I20260620 15:21:58.173043 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:58.162426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47022 (local address 127.0.97.124:40989)
0620 15:21:58.165549 (+ 3123us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:58.165557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:58.165576 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:21:58.165687 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:58.165691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:58.165843 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:58.166013 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:58.166557 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:58.167823 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:58.169804 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:58.170471 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:58.170603 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:58.171165 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:58.171907 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:58.172058 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:58.172598 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 15:21:58.172710 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2986,"thread_start_us":2900,"threads_started":1}
I20260620 15:21:58.199764 23512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:58.179635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.0.97.126:35209)
0620 15:21:58.192806 (+ 13171us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:58.192810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:58.192826 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:21:58.192912 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:58.192916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:58.193048 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:58.193226 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:58.193893 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:58.195230 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:58.197417 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:58.197857 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:58.197991 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:58.198454 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:58.198989 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:58.199114 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:58.199366 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 15:21:58.199458 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13068,"thread_start_us":12978,"threads_started":1}
I20260620 15:21:58.203145 23505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:58.186556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.0.97.124:40989)
0620 15:21:58.186693 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:58.186698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:58.195265 (+ 8567us) server_negotiation.cc:407] Connection header received
0620 15:21:58.195432 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:58.195438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:58.195527 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:58.195620 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:58.196194 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:58.197270 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:58.201329 (+ 4059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:58.201859 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:58.201974 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:58.202448 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:58.202561 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:58.202654 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:58.202832 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 15:21:58.202902 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 15:21:58.204154 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:21:58.179316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.0.97.125:45625)
0620 15:21:58.179469 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 15:21:58.179474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:21:58.193922 (+ 14448us) server_negotiation.cc:407] Connection header received
0620 15:21:58.194109 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:21:58.194114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:21:58.194197 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:21:58.194281 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:21:58.197462 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:58.198600 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:58.200870 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:21:58.201497 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:21:58.201617 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:21:58.202450 (+ 833us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:21:58.202586 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:21:58.202676 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:21:58.203816 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0620 15:21:58.203891 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:21:58.208638 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 23148
I20260620 15:21:58.209826 23210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:58.318979 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 23148
W20260620 15:21:58.338876 23312 connection.cc:535] server connection from 127.0.0.1:46976 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 15:21:58.339546 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 23222
I20260620 15:21:58.340725 23284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:58.453016 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 23222
I20260620 15:21:58.473166 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu pid 23296
I20260620 15:21:58.474254 23358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:21:58.578419 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskRnfwZV/build/asan/bin/kudu with pid 23296
2026-06-20T15:21:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4634 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9340 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (352117 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST